rootupgrade on last node: "CRS-4000: Command StartUpgrade failed" if pre-11.2 user application resource exists (Doc ID 2110614.1)

Last updated on MARCH 03, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Upgrading 11gR2 GI to 12c, rootupgrade.sh fails on last node:

2016-01-13 13:06:50: Executing cmd: /oracle/app/12.1.0/grid/bin/crsctl startupgrade
2016-01-13 13:06:54: Command output:
> Start upgrade invoked..
> CRS-4000: Command StartUpgrade failed, or completed with errors.
>End Command output
2016-01-13 13:06:54: Failed to execute 'crsctl startupgrade'

 

crsd.log

2016-01-13 13:06:54.686: [ CRSPE][1449208128] {2:7633:158} Converting 21 resources...
2016-01-13 13:06:54.686: [ CRSPE][1449208128] {2:7633:158} ...but will skip 3 aliases
2016-01-13 13:06:54.686: [UiServer][1451309376] Sending to PE. ctx= 0x2b9f583c6db0, ClientPID=12454, tint: {0:0:0}
2016-01-13 13:06:54.688: [ CRSPE][1449208128] {2:7633:158} Using ID [55] for attribute: ACL
2016-01-13 13:06:54.690: [ CRSPE][1449208128] {2:7633:158} Using ID [59] for attribute: ACTION_FAILURE_TEMPLATE
....
2016-01-13 13:06:54.783: [ CRSPE][1449208128] {2:7633:158} Using ID [1003] for attribute: USR_ORA_DEBUG
2016-01-13 13:06:54.785: [ CRSPE][1449208128] {2:7633:158} Using ID [1004] for attribute: USR_ORA_DISCONNECT
2016-01-13 13:06:54.787: [ CRSPE][1449208128] {2:7633:158} Using ID [1005] for attribute: USR_ORA_FLAGS
2016-01-13 13:06:54.790: [ CRSD][1449208128] {2:7633:158} Dump State Starting ...
2016-01-13 13:06:54.790: [ CRSD][1449208128] {2:7633:158} State Dump for RTILock
2016-01-13 13:06:54.956: [ default][112192352] First attempt: init CSS context succeeded. ====>>>> crsd crashed and auto restarted
2016-01-13 13:06:54.958: [ clsdmt][120744256]PID for the Process [16968], connkey 1
2016-01-13 13:06:55.914: [ default][120744256] Process environment is not initialized yet!
2016-01-13 13:06:55.914: [ default][112192352] CRS Daemon Starting
2016-01-13 13:06:55.914: [ default][112192352] Attempt to add duplicate debug module CRSUI
2016-01-13 13:06:55.914: [ default][112192352] Attempt to add duplicate debug module CRSCOMM
2016-01-13 13:06:55.914: [ default][112192352] Attempt to add duplicate debug module CRSRTI
2016-01-13 13:06:55.914: [ default][112192352] Attempt to add duplicate debug module CRSMAIN
....
2016-01-13 14:38:29.105: [ CRSPE][356555072] {2:49303:491} Using ID [1003] for attribute: USR_ORA_DEBUG
2016-01-13 14:38:29.107: [ CRSPE][356555072] {2:49303:491} Using ID [1004] for attribute: USR_ORA_DISCONNECT
2016-01-13 14:38:29.109: [ CRSPE][356555072] {2:49303:491} Using ID [1005] for attribute: USR_ORA_FLAGS
2016-01-13 14:38:29.111: [ CRSD][356555072] {2:49303:491} Dump State Starting ...
2016-01-13 14:38:29.111: [ CRSD][356555072] {2:49303:491} State Dump for RTILock
2016-01-13 14:38:29.248: [ default][4088605536] First attempt: init CSS context succeeded.

 

If upgrading to 12.1.0.2, alert.log will show: 

Fri Jan 29 09:56:04 2016
Errors in file /oracle/app/11.2.0/base/diag/crs/us1us0174/crs/trace/crsd.trc (incident=1):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /oracle/app/11.2.0/base/diag/crs/us1us0174/crs/incident/incdir_1/crsd_i1.trc

Errors in file /oracle/app/11.2.0/base/diag/crs/us1us0174/crs/trace/crsd.trc (incident=2):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /oracle/app/11.2.0/base/diag/crs/us1us0174/crs/incident/incdir_2/crsd_i2.trc

 

crsd.trc 

2016-01-29 09:58:21.203096 : CRSPE:1669445952: {2:60623:187} Converting 26 resources...
2016-01-29 09:58:21.203116 : CRSPE:1669445952: {2:60623:187} ...but will skip 6 aliases
2016-01-29 09:58:21.206000 : CRSPE:1669445952: {2:60623:187} Using ID [55] for attribute: ACL
2016-01-29 09:58:21.208621 : CRSPE:1669445952: {2:60623:187} Using ID [59] for attribute: ACTION_FAILURE_TEMPLATE
..
2016-01-29 09:58:21.323888 : CRSPE:1669445952: {2:60623:187} Using ID [1005] for attribute: USR_ORA_FLAGS
CLSB:1669445952: Oracle Clusterware infrastructure error in CRSD (OS PID 30600): Fatal signal 11 has occurred in program crsd thread 1669445952; nested signal count is 1

 

crsd.bin call stack:  

[06]: __sighandler []
[07]: _ZStlsIcSt11char_traitsIcESaIcEERSt13basic_ostreamIT_T0_ES7_RKSbIS4_S5_T1_E []
[08]: _ZNK6cls_pe8Resource20getAttributeName4OcrERKSsRSsb []
[09]: _ZNK6cls_pe8Resource18getValuesAsOcrKeysERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEESA_RS8_bb []
[10]: _ZNK6cls_pe8Resource18getValuesAsOcrKeysEbbRSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEEbb []
[11]: _ZNK6cls_pe8Resource28getOcrKeyValuesIn12gR1FormatERSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE []
[12]: _ZN6cls_pe23UpgradeTriggerOperation18convertResources12ERKNS_9DataModelE []
[13]: _ZN6cls_pe23UpgradeTriggerOperation35deletePrevNewResourceTreeCallback12EPN7cls_ocr16KeyDeleteMessageE []
[14]: _ZN6cls_pe17ResourceOperation13MessageBroker15callbackHandlerEPN3cls7MessageE []
[15]: _ZN3cls11ThreadModel12processQueueEP7sltstid []
[16]: _ZN3cls11ThreadModel5runTMEPv []
[17]: _ZN13CLS_Threading13CLSthreadMain8cppStartEPv []
[18]: start_thread []

  

"crsctl stat res -f" output for the offending resource:  

NAME=SIOPAD_US1US0174
TYPE=application
CRS-601: Internal error
RC: 5, File: clsStatParser.cpp, Line: 2755
CRS-4000: Command Status failed, or completed with errors.

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms