[OCI] : GI Patching Failed With 'This is an internal error' 'Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191)' (Doc ID 2784501.1)

Last updated on JUNE 22, 2021

Applies to:

Symptoms

OCI RAC Setup.
GI patching failed and describe job shows , below details:

Job details
----------------------------------------------------------------
ID: ffa4db70-68df-4cb6-b427-da71991a2ba0
Description: Server Patching
Status: Failure
Created: June 13, 2021 12:57:31 AM CDT
Progress: 42%
Message: DCS-10001:Internal error encountered: Failure : failed to apply 32545013 on /u01/app/19.0.0.0/grid.
Cause: This is an internal error.
Action: Contact Oracle Support Services for assistance.

Task Name Start Time End Time Status
------------------------------------------------------------------------ ----------------------------------- ----------------------------------- ----------
Server Patching June 13, 2021 12:57:31 AM CDT June 13, 2021 1:10:21 AM CDT Failure
Pre-operations for Server Patching June 13, 2021 12:57:31 AM CDT June 13, 2021 1:00:33 AM CDT Success
Server Patching June 13, 2021 1:00:33 AM CDT June 13, 2021 1:10:21 AM CDT Failure
task:TaskSequential_2226 June 13, 2021 1:00:33 AM CDT June 13, 2021 1:10:21 AM CDT Failure
task:TaskSequential_2285 June 13, 2021 1:05:49 AM CDT June 13, 2021 1:10:21 AM CDT Failure
task:TaskSequential_2286 June 13, 2021 1:05:49 AM CDT June 13, 2021 1:10:21 AM CDT Failure
apply clusterware patch[NODE1] June 13, 2021 1:05:49 AM CDT June 13, 2021 1:10:20 AM CDT Failure

From /var/opt/oracle/dcs/dcs-agent.0.0.log

...
[2021-06-13 02:26:13,953] SEVERE [apply clusterware patch[NODE1] : JobId=21ea4fd2-fa12-4ade-83fc-68f8ec5be52d] com.oracle.dcs.commons.jrpc.JsonRequestProcessor: RPC request invocation failed on request: {"classz":"com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions",
"method":"applyPatchWithTag",
"params":[{"type":"java.lang.String",
"value":"gi"},
{"type":"java.lang.String",
"value":""},
{"type":"java.lang.String",
"value":"19.11.0.0.210420"}],
"revertable":false,
"threadId":342}
DcsException{errorHttpCode=InternalError, msg=Internal error encountered: Failure : failed to apply 32545013 on /u01/app/19.0.0.0/grid. , msgId=10001, causedBy=null}
at com.oracle.dcs.commons.exception.DcsException$Builder.build(DcsException.java:69)
at com.oracle.dcs.agent.task.patching.OpatchAutoApplyTask.run(OpatchAutoApplyTask.java:265)
at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyPatchUsingOpatchAuto(BmsPatchingActions.java:472)
at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyPatchActions(BmsPatchingActions.java:320)
at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyGiPatches(BmsPatchingActions.java:1012)
....
....
[2021-06-13 02:26:13,954] FINE [apply clusterware patch[NODE1] : JobId=21ea4fd2-fa12-4ade-83fc-68f8ec5be52d] com.oracle.dcs.agent.task.TaskZJsonRpcExt$$Lambda$631/662667351: Timeout task completed. Task result is: {
"data" : null,
"error" : "DCS-10001:Internal error encountered: Failure : failed to apply 32545013 on /u01/app/19.0.0.0/grid. ",
"requestThreadId" : 342,
"dataType" : null
}
[2021-06-13 02:26:13,954] FINE [apply clusterware patch[NODE1] : JobId=21ea4fd2-fa12-4ade-83fc-68f8ec5be52d] com.oracle.dcs.agent.zk.DCSZooKeeper: DCS node id is - node_0
[2021-06-13 02:26:13,954] FINE [apply clusterware patch[NODE1] : JobId=21ea4fd2-fa12-4ade-83fc-68f8ec5be52d] com.oracle.dcs.agent.task.TaskZJsonRpc: Task[TaskZJsonRpcExt_3842] RPC request 'Local:node_0@applyPatchWithTag()' completed: Failure
[2021-06-13 02:26:13,954] FINE [apply clusterware patch[NODE1] : JobId=21ea4fd2-fa12-4ade-83fc-68f8ec5be52d] com.oracle.dcs.agent.task.TaskZJsonRpcExt: Task[id: TaskZJsonRpcExt_3842, TaskName: apply clusterware patch[NODE1]] got exception:
com.oracle.dcs.commons.exception.DcsTaskFailureException: DCS-10001:Internal error encountered: Failure : failed to apply 32545013 on /u01/app/19.0.0.0/grid.
at com.oracle.dcs.agent.task.TaskZJsonRpc.processJsonResponse(TaskZJsonRpc.java:62)

From systemconfig2021-06-13_03-53-45AM.log

NODE_STATUS::NODE2:EFAIL
The result of cluvfy command contain EFAIL NODE_STATUS::NODE2:EFAIL
2021-06-13 03:53:52,696 SEVERE [1] com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator - Stacktrace :: oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Unable to determine if "/u01/app/19.0.0.0/grid" is a shared oracle home.
Failed:
Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191)
PRVF-4009 : User equivalence is not set for nodes: :NODE2
Verification will proceed with nodes: NODE1
...
Caused by: java.lang.Exception: The result of cluvfy command contain EFAIL NODE_STATUS::NODE2:EFAIL
at oracle.dbsysmodel.driver.sdk.util.OsysUtility.isSharedPath(OsysUtility.java:151)
... 7 more
2021-06-13 03:53:52,698 SEVERE [1] com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator - Failure reason::java.lang.Exception: The result of cluvfy command contain EFAIL NODE_STATUS::NODE2:EFAIL

From bootstrap2021-06-13_03-49-30AM.log

2021-06-13 03:49:48,179 INFO [1] com.oracle.glcm.patch.auto.db.util.PerlBootstrapPlugin - PerlPath is null
2021-06-13 03:49:48,317 SEVERE [1] com.oracle.glcm.patch.auto.db.util.SRVMBootstrapPlugin - OPATCHAUTO-72082: Multiple system patch supplied.
OPATCHAUTO-72082: opatchauto cannot process multiple system patches as part of same session.
OPATCHAUTO-72082: Please remove one system patch from the patch location.
2021-06-13 03:49:48,317 INFO [1] com.oracle.glcm.patch.auto.db.util.BootstrapHandler - The bootstrap execution result is : [BootstrapResult [result=SUCCESS, pluginType=PERL, errorMessage=null, errorCode=null], BootstrapResult [result=FAILED, pluginType=SRVM, errorMessage=OPATCHAUTO-72082: Multiple system patch supplied.
OPATCHAUTO-72082: opatchauto cannot process multiple system patches as part of same session.
OPATCHAUTO-72082: Please remove one system patch from the patch location., errorCode=72058]]
2021-06-13 03:49:48,322 SEVERE [1] com.oracle.glcm.patch.auto.db.util.BootstrapHandler - OPATCHAUTO-72083: Performing bootstrap operations failed.
OPATCHAUTO-72083: The bootstrap execution failed because OPATCHAUTO-72082: Multiple system patch supplied.
OPATCHAUTO-72082: opatchauto cannot process multiple system patches as part of same session.
OPATCHAUTO-72082: Please remove one system patch from the patch location..
OPATCHAUTO-72083: Fix the reported problem and re-run opatchauto.
Type :quit to exit Vim

 O cluvfy comp nodereach may give below error

$Grid_home/bin/cluvfy comp nodereach -n <other node>

PRCT-1011 : Failed to run "crsctl". Detailed error: CRS-4654: Server 'NODE01-vip' could not be found.,CRS-4000: Command Get failed, or completed with errors.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.