Running rootcrs.pl -postpatch Causes Node Reboot and CRS Fails To Start (Doc ID 2013910.1)

Last updated on NOVEMBER 01, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Environment:
Exadata Quarter rack with two DB nodes three cell nodes
Image 12.1.1.1.2

Followed Doc ID 1591616.1 to upgrade GI & RDBMS from 12.1.0.2 BP 1 to 12.1.0.2 BP 6

After the upgrade patching is completed, the last step of executing rootcrs -postpatch caused the node to reboot continuously.


Step 5 Manual Steps for Apply/Rollback Patch
As the root user execute:
# <GI_HOME>/crs/install/rootcrs.pl -postpatch  <<< failing here

Diagnosis:

- Disabled crs to avoid HAS from rebooting the node.

- Now the node boots up but CRS fails to start.


Errors reported:
================
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2015/05/24 20:22:46 CLSRSC-117: Failed to start Oracle Clusterware stack

Died at /u01/app/12.1.0.2/grid/crs/install/crspatch.pm line 880.
================

crspatch.pm line 880 tries to start the crs stack:


====From 12.1.0.2 crspath.pm script:
   877   modifyOC4JFiles($patchGIHome);
    878
    879   trace("Starting Oracle Clusterware stack for rolling patch ...");
    880   startFullStack($patchGIHome) || die(dieformat(117));
    881
    882   clscfgPatch($patchGIHome) || die(dieformat(180, 'clscfg -patch', '-1'));
    883   stopRollingPatch($patchGIHome) ||  die(dieformat(431));
    884
==============

- Checked the crs alert log & it was reporting issues with cssd:

===
Errors in file /u01/app/oracle/diag/crs/<nodename>/crs/trace/ocssd.trc  (incident=<###>):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/crs/<nodename>/crs/incident/incdir_<###>/ocssd_i<###>.trc
===

-  The cssd incident trace ocssd_i<###>.trc :
    a) "signal 11"
    ====
    2015-05-24 23:27:43.235684*:CLSB:1520625216: Oracle Clusterware infrastructure error in OCSSD (OS PID 23011): Fatal signal 11 has occurred in program ocssd thread 1520625216; nested signal count is 1
    ====
    b) Stack below
===
Error: CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
[00]: dbgexProcessError [diag_dde]
[01]: dbgeExecuteForError [diag_dde]
[02]: dbgePostErrorDirect [diag_dde]
[03]: clsdAdrPostError []<-- Signaling
[04]: clsbSigErrCB []
[05]: skgesig_sigactionHandler []
[06]: __sighandler []
[07]: gipcUnlockF []
[08]: gipcWaitProcessEndpoint []
[09]: gipcInternalWaitEpoll []
[10]: gipcInternalWait []
[11]: gipcWaitF []
[12]: gipcInternalRecvSync []
[13]: gipcRecvSyncF []
[14]: clsgpnpm_gipcGets []
[15]: clsgpnpm_receiveMsg []
[16]: clsgpnpm_exchange []
[17]: clsgpnp_profileCallUrlInt []
[18]: clsgpnp_getProfileEx []
[19]: clsgpnp_getProfile []
[20]: clssscGPNPInit []
[21]: clssscInitGlobalCTX []
[22]: clssscmain []
[23]: clsbCMain []
[24]: main []
[25]: __libc_start_main []

 

Changes

Upgrade of GRID from 12.1.0.2 BP 1 to 12.1.0.2 BP 6

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms