Wrong DiscoveryString /dev/*: rootupgrade.sh/root.sh hangs: Check OCR key using ocrdump

(Doc ID 1916106.1)

Last updated on MAY 24, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading to 12.1.0.2 Grid Infrastructure, rootupgrade.sh hangs on node1 after the following is printed on screen: 

2014-08-09 23:37:07 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.               ====>> hangs here

 

<NEW_GI_HOME>/cfgtoollogs/crsconfig/rootcrs_<node>_<timestamp>.log 

2014-08-09 23:35:37: CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2014-08-09 23:35:37: Executing cmd: /opt/oracle/grid12/bin/crsctl stop crs -f
2014-08-09 23:37:07: Command output:
> CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cadbpoc1'
> CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
....
> CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode1' has completed
> CRS-4133: Oracle High Availability Services has been stopped.
>End Command output
....
2014-08-09 23:37:11: Current first node: racnode1
2014-08-09 23:37:11: Trying to get the value of key: SYSTEM.rootcrs.checkpoints.firstnode in OCR.
2014-08-09 23:37:11: setting ORAASM_UPGRADE to 1
2014-08-09 23:37:11: Check the existence of key pair with key name: SYSTEM.rootcrs.checkpoints.firstnode in OCR.
2014-08-09 23:37:11: setting ORAASM_UPGRADE to 1
2014-08-09 23:37:11: Invoking "/opt/oracle/grid12c/bin/cluutil -exec -keyexists -key checkpoints.firstnode"
2014-08-09 23:37:11: trace file=/opt/oracle/base/crsdata/racnode1/crsconfig/cluutil2.log
2014-08-09 23:37:11: Running as user oracle: /opt/oracle/grid12c/bin/cluutil -exec -keyexists -key checkpoints.firstnode
2014-08-09 23:37:11: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /opt/oracle/grid12c/bin/cluutil -exec -keyexists -key checkpoints.firstnode '
2014-08-09 23:37:12: Removing file /tmp/fileZAtDSM
2014-08-09 23:37:12: Successfully removed file: /tmp/fileZAtDSM
2014-08-09 23:37:12: pipe exit code: 256
2014-08-09 23:37:12: /bin/su exited with rc=1

2014-08-09 23:37:12: oracle.ops.mgmt.rawdevice.OCRException: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]  ====>> this message can be ignored

2014-08-09 23:37:12: Cannot get OCR key with CLUUTIL, try using OCRDUMP.
2014-08-09 23:37:12: Check OCR key using ocrdump                                  ====>> hangs here

 

<ORACLE_BASE>/diag/crs/wirac1f/crs/trace/ocrdump_<pid>.trc

OCRASM: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=29701, loc=kgfokge

2014-08-09 23:37:20.562899 : OCRASM: ASM Error Stack : ORA-29701: unable to connect to Cluster Synchronization Service

2014-08-09 23:37:20.562913 : OCRASM: proprasmo: ASM instance is down. Proceed to open the file in dirty mode.
CLWAL: clsw_Initialize: Error [32] from procr_init_ext
CLWAL: clsw_Initialize: Error [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]] from procr_init_ext
2014-08-09 23:37:20.565792 : GPNP: clsgpnpkww_initclswcx: [at clsgpnpkww.c:351] Result: (56) CLSGPNP_OCR_INIT. (:GPNP01201:)Failed to init CLSW-OLR context. CLSW Error (3): CLSW-3: Error in the cluster registry (OCR) layer. [32] [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]]
2014-08-09 23:37:20.586458 : OCRASM: proprasmo: Error [13] in opening the GPNP profile. Try to get offline profile
WARNING: Using brute force method to determine the size of /dev/network_throughput.
There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
WARNING: Using brute force method to determine the size of /dev/network_latency.
There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
WARNING: Using brute force method to determine the size of /dev/cpu_dma_latency.
There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
WARNING: Using brute force method to determine the size of /dev/usbmon2.
There will be performance issues. Please check configuration to determine the cause for the failure of ioctl                             ====>> hangs here


Output of "<GI_HOME>/bin/gpnptool get":

<orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/*" SPFile="+DATA/eotcs3/asmparameterfile/registry.253.854838923"/>

 

The issue can also happen on non-first node (second etc) during fresh install.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms