My Oracle Support Banner

OCI : Server Patching Failed with Error 'DCS-10001:Internal error encountered: Failure : failed to apply ' 'Could not get node role for node' 'InventoryDoesNotExistException' (Doc ID 2679970.1)

Last updated on JUNE 12, 2020

Applies to:

Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.

Symptoms

2 node RAC DB system in OCI env.

Applying latest PSU to GI (Server Patching) home , got failed with below message :

#/opt/oracle/dcs/bin/dbcli describe-job -i f4ab5c87-9339-448c-8cc4-fe02ba567461

Job details
----------------------------------------------------------------
ID: f4ab5c87-9339-448c-8cc4-fe02ba567461
Description: Server Patching
Status: Failure
Created: June 6, 2020 10:31:59 PM CDT
Progress: 42%
Message: DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.

Task Name Start Time End Time Status
------------------------------------------------------------------------ ----------------------------------- ----------------------------------- ----------
Server Patching June 6, 2020 10:31:59 PM CDT June 6, 2020 10:41:31 PM CDT Failure
Pre-operations for Server Patching June 6, 2020 10:31:59 PM CDT June 6, 2020 10:32:13 PM CDT Success
Server Patching June 6, 2020 10:32:13 PM CDT June 6, 2020 10:41:31 PM CDT Failure
task:TaskSequential_6811 June 6, 2020 10:32:13 PM CDT June 6, 2020 10:41:31 PM CDT Failure
task:TaskSequential_6865 June 6, 2020 10:36:21 PM CDT June 6, 2020 10:41:31 PM CDT Failure
task:TaskSequential_6866 June 6, 2020 10:36:21 PM CDT June 6, 2020 10:41:30 PM CDT Failure
apply clusterware patch[NODE51] June 6, 2020 10:36:21 PM CDT June 6, 2020 10:41:30 PM CDT Failure

From /opt/oracle/dcs//log/dcs-agent.log

2020-06-06 01:08:42,985 DEBUG [Patch conflict check[NODE51] : JobId=2de924e7-b568-4b6e-bb65-bc01319aa5e1] [] c.o.d.c.u.CommonsUtils: Output :

OPatchauto session is initiated at Sat Jun 6 01:08:13 2020

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2020-06-06_01-08-18AM.log.

OPatchAuto failed.

OPatchauto session completed at Sat Jun 6 01:08:42 2020
Time taken to complete the session 0 minute,
36 seconds

opatchauto failed with error code 42

From systemconfig2020-06-06_01-08-18AM.log.

2020-06-06 01:08:21,107 INFO [1] oracle.dbsysmodel.driver.sdk.util.OsysUtility - Output message :::
2020-06-06 01:08:41,533 CONFIG [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - shared: false
2020-06-06 01:08:41,633 CONFIG [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - the node role for host: NODE51 is: hub
2020-06-06 01:08:41,754 CONFIG [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - Could not get node role for node: NODE52 with error PRCI-1152 : failed to retrieve the configured node role of the node NODE52
PRCT-1011 : Failed to run "crsctl". Detailed error: CRS-4654: Server 'NODE52' could not be found.,CRS-4000: Command Get failed, or completed with errors.
2020-06-06 01:08:42,343 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver - dbList.size: 1
2020-06-06 01:08:42,383 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver - isAdminPolicyDBMixedSetup: false
2020-06-06 01:08:42,424 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - homeType: crs
2020-06-06 01:08:42,425 FINE [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - Hometype of:e3a990fb-3ba7-4507-9ad2-9cd0150b8caf set to crs
2020-06-06 01:08:42,425 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - homeType: crs
2020-06-06 01:08:42,426 FINE [1] com.oracle.glcm.patch.auto.db.product.driver.crs.AbstractCrsProductDriver - Hometype of:910f106b-44fd-4565-8761-fb8dade497f7 set to crs

O Verifying node role

crsctl stat res -t
crsctl get node role status -all

from node 1:

[root@NODE51 bin]# ./crsctl get node role status -all
Node 'NODE51' active role is 'hub'
CRS-4654: Server 'NODE52' could not be found.
CRS-4000: Command Get failed, or completed with errors.

from node 2:

[root@NODE52 bin]# ./crsctl get node role status -all
CRS-4654: Server 'NODE51' could not be found.
CRS-4000: Command Get failed, or completed with errors.

From /opt/oracle/dcs/log/dcs-agent-2020-06-06.log

2020-06-06 22:41:30,966 ERROR [apply clusterware patch[NODE51] : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.c.j.JsonRequestProcessor: RPC request invocation failed on request: {"classz":"com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions",
"method":"applyPatchWithTag",
"params":[{"type":"java.lang.String",
"value":"gi"},
{"type":"java.lang.String",
"value":""},
{"type":"java.lang.String",
"value":"12.2.0.1.200114"}],
"revertable":false,
"threadId":385}
! com.oracle.dcs.commons.exception.DcsException: DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.
! at com.oracle.dcs.commons.exception.DcsException$Builder.build(DcsException.java:69)
! at com.oracle.dcs.agent.task.patching.OpatchAutoApplyTask.run(OpatchAutoApplyTask.java:244)
! at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyPatchUsingOpatchAuto(BmsPatchingActions.java:400)
! at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyPatchActions(BmsPatchingActions.java:287)
! at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyGiPatches(BmsPatchingActions.java:927)
! at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyGiPatch(BmsPatchingActions.java:803)
! at com.oracle.dcs.agent.rpc.service.patching.BmsPatchingActions.applyPatchWithTag(BmsPatchingActions.java:91)
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
! at java.lang.reflect.Method.invoke(Unknown Source)
! at com.oracle.dcs.commons.jrpc.JsonRequestProcessor.invokeRequest(JsonRequestProcessor.java:32)
.............
2020-06-06 22:41:30,966 DEBUG [apply clusterware patch[NODE51] : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.a.z.DCSZooKeeper: DCS node id is - node_0
2020-06-06 22:41:30,966 DEBUG [apply clusterware patch[NODE51] : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.a.t.TaskZJsonRpc: Task[TaskZJsonRpcExt_6867] RPC request 'Local:node_0@applyPatchWithTag()' completed: Failure
2020-06-06 22:41:30,967 DEBUG [apply clusterware patch[NODE51] : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.a.t.TaskZJsonRpcExt: Task[id: TaskZJsonRpcExt_6867, TaskName: apply clusterware patch[NODE51]] got exception:
! com.oracle.dcs.commons.exception.DcsTaskFailureException: DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.
! at com.oracle.dcs.agent.task.TaskZJsonRpc.processJsonResponse(TaskZJsonRpc.java:62)
! at com.oracle.dcs.agent.task.TaskZJsonRpcExt.callInternal(TaskZJsonRpcExt.java:60)
! at com.oracle.dcs.agent.task.TaskZJsonRpc.call(TaskZJsonRpc.java:198)
! at com.oracle.dcs.agent.task.TaskZJsonRpc.call(TaskZJsonRpc.java:30)
! at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:95)
! at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:52)
! at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:39)
! at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:10)
! at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:95)
.....
2020-06-06 22:41:30,967 DEBUG [apply clusterware patch[NODE51] : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.c.t.r.ServiceJobReport: process TaskDetail this.jobid= f4ab5c87-9339-448c-8cc4-fe02ba567461 d.jobid=f4ab5c87-9339-448c-8cc4-fe02ba567461 d.taskid=TaskZJsonRpcExt_6867 this.msg=null d.res=DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid. d.subtasks.isempty=true d.status=Failure

2020-06-06 22:41:31,143 DEBUG [Server Patching : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.c.t.TaskDetail: set task result as DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.

2020-06-06 22:41:31,202 DEBUG [Server Patching : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.c.t.TaskDetail: set task result as DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.
2020-06-06 22:41:31,202 INFO [Server Patching : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.a.t.TaskZLockWrapper: Task[id: f4ab5c87-9339-448c-8cc4-fe02ba567461, TaskName: Server Patching] result: DCS-10001:Internal error encountered: Failure : failed to apply 30593149 on /u01/app/12.2.0.1/grid.
2020-06-06 22:41:31,202 INFO [Server Patching : JobId=f4ab5c87-9339-448c-8cc4-fe02ba567461] [] c.o.d.a.t.TaskZLockWrapper: Task[id: f4ab5c87-9339-448c-8cc4-fe02ba567461, TaskName: Server Patching] completed: Failure

later
-------
75a41669-caf1-4b40-8c8a-0e11def4248f Server Patching Prechecks June 6, 2020 10:49:20 PM CDT Success
.................
2020-06-06 22:51:45,127 ERROR [Patch conflict check[NODE51] : JobId=75a41669-caf1-4b40-8c8a-0e11def4248f] [] c.o.d.c.u.CommonsUtils: Jun 06,
2020 10:51:45 PM com.oracle.glcm.patch.auto.OPatchAuto main
SEVERE: OPatchAuto failed.
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: oracle.sysman.oii.oiii.OiiiInventoryDoesNotExistException: The inventory pointed at location /u01/app/oracle/oraInventory is not valid



Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.