GI rootupgrade.sh Fails due to Upper Case HOSTING_MEMBERS in OCR (Doc ID 1592365.1)

Last updated on APRIL 08, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.1.0.7 and later
Information in this document applies to any platform.

Symptoms


Upgrading Oracle Clusterware from 11.1.0.7.7 to 11.2.0.3, rootupgrade.sh fails on node1

# <NEW_GI_HOME>/rootupgrade.sh
Performing root user operation for Oracle 11g
....
Replacing Clusterware entries in inittab
Oracle Grid Infrastructure stack start initiated but failed to complete at /u01/app/grid/product/11.2.0.3/grid_1/crs/install/crsconfig_lib.pm line 11486.
/u01/app/grid/product/11.2.0.3/grid_1/perl/bin/perl -I/u01/app/grid/product/11.2.0.3/grid_1/perl/lib -I/u01/app/grid/product/11.2.0.3/grid_1/crs/install /u01/app/grid/product/11.2.0.3/grid_1/crs/install/rootcrs.pl execution failed
2013-10-10 16:55:46: Executing cmd: /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl start resource ora.crsd -init
2013-10-10 16:55:48: Command output:
> CRS-2672: Attempting to start 'ora.crsd' on 'srvtst1'
> CRS-2676: Start of 'ora.crsd' on 'srvtst1' succeeded ====>> started but went down shortly
>End Command output
....
2013-10-10 16:55:50: Executing cmd: /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl check crs
2013-10-10 16:55:50: Command output:
> CRS-4638: Oracle High Availability Services is online
> CRS-4535: Cannot communicate with Cluster Ready Services ====>> can't contact to crsd
> CRS-4529: Cluster Synchronization Services is online
> CRS-4534: Cannot communicate with Event Manager
>End Command output
2013-10-10 16:55:50: Running /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl check cluster -n srvtst1
2013-10-10 16:55:50: Executing cmd: /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl check cluster -n srvtst1
2013-10-10 16:55:50: Checking the status of cluster
2013-10-10 16:55:55: Executing cmd: /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl check cluster -n srvtst1
..
..
2013-10-10 17:06:23: Executing cmd: /u01/app/grid/product/11.2.0.3/grid_1/bin/crsctl check cluster -n srvtst1
2013-10-10 17:06:23: Checking the status of cluster
2013-10-10 17:06:28: ###### Begin DIE Stack Trace ######
2013-10-10 17:06:28: Package File Line Calling
2013-10-10 17:06:28: --------------- -------------------- ---- ----------
2013-10-10 17:06:28: 1: main rootcrs.pl 375 crsconfig_lib::dietrap
2013-10-10 17:06:28: 2: crsconfig_lib crsconfig_lib.pm 11486 main::__ANON__
2013-10-10 17:06:28: 3: crsconfig_lib crsconfig_lib.pm 1296 crsconfig_lib::wait_for_stack_start
2013-10-10 17:06:28: 4: crsconfig_lib crsconfig_lib.pm 1171 crsconfig_lib::start_cluster
2013-10-10 17:06:28: 5: main rootcrs.pl 803 crsconfig_lib::perform_start_cluster
2013-10-10 17:06:28: ####### End DIE Stack Trace #######
2013-10-10 16:55:48.696
[crsd(7573)]CRS-1012:The OCR service started on node srvtst1.
2013-10-10 16:55:49.706
[evmd(7581)]CRS-1401:EVMD started on node srvtst1.
2013-10-10 16:55:52.201
[crsd(7573)]CRS-1201:CRSD started on node srvtst1.
2013-10-10 16:55:59.646
[ohasd(7159)]CRS-2765:Resource 'ora.crsd' has failed on server 'srvtst1'.
2013-10-10 16:55:52.903: [ CRSRES][52] startup = 1
2013-10-10 16:55:52.998: [ CRSD][54] Dump State Starting ...
2013-10-10 16:55:52.998: [ CRSD][54] State Dump for RTILock
2013-10-10 16:55:52.998: [ CRSD][54] LockName:_CRS_RDB::ThreadId:56
2013-10-10 16:55:52.998: [ CRSD][54] LockName:_ORA_CRS_BOOT::ThreadId:52
2013-10-10 16:55:52.998: [ CRSD][54] LockName:_ORA_CRS_MEMBER_srvtst1::ThreadId:52
2013-10-10 16:55:52.998: [ CRSD][54] State Dump for Timer
2013-10-10 16:55:52.998: [ CRSD][54] Timer map size=0
2013-10-10 16:55:53.012: [ CRSPE][54] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
------------- RESOURCES:

------------- TYPES:

------------- SERVERS:

------------- SERVER POOLS:

2013-10-10 16:55:53.012: [ CRSPE][54] Dumping ICE contents...:ICE operation count: 0
2013-10-10 16:55:53.012: [ CRSD][54] Dump State Done.
2013-10-10 16:55:53.051: [ CRSRES][56] Attempting to start `ora.PUAT.PUAT1.inst` on member `srvtst1`
2013-10-10 16:55:53.076: [ CRSRES][57] Attempting to start `ora.STGREP.STGREP1.inst` on member `srvtst1`
2013-10-10 16:55:53.126: [ CRSRES][58] Attempting to start `ora.SUAT.SUAT1.inst` on member `srvtst1`
2013-10-10 16:55:53.232: [ CRSRES][55] Attempting to start `ora.PRDREP.PRDREP1.inst` on member `srvtst1`
2013-10-10 16:55:53.359: [ CRSRES][53] Attempting to stop `ora.srvtst1.vip` on member `srvtst2`
2013-10-10 16:55:54.149: [ CRSRES][53] Stop of `ora.srvtst1.vip` on member `srvtst2` succeeded.
2013-10-10 16:55:54.227: [ CRSRES][53] Attempting to start `ora.srvtst1.vip` on member `srvtst1`
2013-10-10 16:55:55.182: [UiServer][51] CS(600000000122fdc0)set Properties ( ,6000000002483b80)
2013-10-10 16:55:55.219: [UiServer][51] CS(600000000122fef0)set Properties ( oracle,6000000001a00fe0)
2013-10-10 16:55:55.219: [UiServer][51] crs_qstat_getfmsg: clsc err 10 on fixed read

2013-10-10 16:55:55.220: [UiServer][51] crs_qstat_process: error in mesg read in qstat server
2013-10-10 16:55:55.220: [UiServer][51] processQstatRequest: Closed: remote end failed/disc.
2013-10-10 16:55:59.870: [ default][1] First attempt: init CSS context succeeded.

 

2013-10-10 16:55:48
CRSD REBOOT
Assertion failed: which<listv.size(), file caa_Resource.cpp, line 1471
2013-10-10 16:55:52
CRSD handling signal 6
Dumping CRSD state
2013-10-10 16:55:52
Dumping CRSD stack trace
2013-10-10 16:55:53


----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------

sclssutl_sigdump()+ call kgdsdst() 60000000000682C0 ?
1584 000000003 ?
9FFFFFFFFF52EE48 ?
C00000001237E0D0 ?
000000541 ?
9FFFFFFFFF51C438 ?
sclssutl_signalhand call sclssutl_sigdump() 9FFFFFFFFF5A9F10 ?
ler()+240 000000006 ?
FFFFFFFFFFFFFFFF ?
<kernel> call sclssutl_signalhand 000000006 ? 000000006 ?
ler() 9FFFFFFFFF5A2E48 ?
C000000000000187 ?
__lwp_kill()+48 signal <kernel> 9FFFFFFF5C967400 ?
100000006 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ?
__pthread_kill()+25 call __lwp_kill() 000000000 ? 000000000 ?
12 000000000 ? 000000000 ?
_raise()+224 call __pthread_kill() 6000000002229800 ?
000000006 ?
abort()+544 call _raise() 000000006 ?
9FFFFFFFFF2ECE98 ?
C00000000000038A ?
_assert()+608 call abort() 00000000C ?
C000000000000690 ?
C0000000003C3F90 ?
_ZN3CAA12relocateth call _assert() 000000000 ? 000000031 ?
isERSt6vectorISt4pa 9FFFFFFF5C96C2FB ?
irISsSsESaIS1_ISsSs 000000000 ?
EEERNS_8ReporterEi(
)+192
_ZN3CAA8Resource5pl call _ZN3CAA12relocateth 9FFFFFFF5C96C660 ?
aceEiRKSsRNS_8Ident isERSt6vectorISt4pa 9FFFFFFF5C9703C0 ?
ityES2_bSt3setISsSt irISsSsESaIS1_ISsSs 000000000 ?
4lessISsESaISsEERS5 EEERNS_8ReporterEi( 9FFFFFFF5C96C660 ?
_ISsS6_ISsESaISsEER ) 9FFFFFFF5C9703C0 ?
NS_8ReporterE()+296 000000000 ?
96 9FFFFFFF5C96D980 ?
_ZN3CAA8Resource11S call _ZN3CAA8Resource5pl 000000006 ?
tartupWork7executeE aceEiRKSsRNS_8Ident 9FFFFFFF5C970128 ?
RSt3mapISsiSt4lessI ityES2_bSt3setISsSt 9FFFFFFF5C970D00 ?
SsESaISt4pairIKSsiE 4lessISsESaISsEERS5 9FFFFFFF5C970D00 ?
EERSt6vectorISsSaIS _ISsS6_ISsESaISsEER 000000001 ?
sEERS5_()+6432 NS_8ReporterE() 9FFFFFFF5C9701B0 ?
9FFFFFFF5C9701F0 ?
9FFFFFFF5C9703C0 ?
_ZN3CAA8Resource15p call _ZN3CAA8Resource11S 000000006 ?
rocesslistStubEPv() tartupWork7executeE 9FFFFFFF5C970D00 ?
+1200 RSt3mapISsiSt4lessI 600000000189C6C0 ?
SsESaISt4pairIKSsiE 600000000189C6C8 ?
EERSt6vectorISsSaIS 60000000023C0C70 ?
sEERS5_()
_ZN13CLS_Threading1 call _ZN3CAA8Resource15p 60000000023C0C70 ?
3CLSthreadMain8cppS rocesslistStubEPv() 60000000023C0C70 ?
tartEPv()+96 60000000023C0C70 ?
60000000013EBCD0 ?
__pthread_bound_bod call _ZN13CLS_Threading1 60000000012716B0 ?
y()+400 3CLSthreadMain8cppS 60000000012716B0 ?
tartEPv() 60000000012716B0 ?
000000000 ? 000000000 ?
000000000 ?





Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms