My Oracle Support Banner

rootupgrade.sh Fails as CSSD Does not Start in Cloned Environment (Doc ID 1611318.1)

Last updated on JANUARY 02, 2020

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading from 10.2.0.5 CRS to a cloned 11.2.0.3 GI, rootupgrade.sh fails on node1

2013-12-31 15:05:40: The active version of the Oracle Clustereware is '10 2 0 5 0'
2013-12-31 15:05:40: Skipping clusterguid fetch for 10.2.0.5.0
2013-12-31 15:05:40: Executing ocrcheck to get ocrid
2013-12-31 15:05:51: Executing cmd: /<11.2.0.3_GI_HOME>/bin/oifcfg iflist -p -n
2013-12-31 15:05:51: Command output:
> <PUBLIC_INTERFACE> 10.xxx.xxx.16 PRIVATE 255.xxx.xxx.240
> <PRIVATE_INTERFACE> 172.xx.x.16 PRIVATE 255.xxx.xxx.240
....
....
2013-12-31 15:05:51: Earlier version Oracle Grid Infrastructure is running
2013-12-31 15:05:51: Executing cmd: /<10.2.0_GI_HOME>/bin/oifcfg getif
2013-12-31 15:05:51: Command output:
> <PUBLIC_INTERFACE> 10.xxx.xxx.16 global public
PRIVATE_INTERFACE> 172.xx.x.16 global cluster_interconnect
>End Command output
2013-12-31 15:05:51: ---Got oifcfg out (/<10.2.0_GI_HOME>/oifcfg getif):
2013-12-31 15:05:51: <PUBLIC_INTERFACE> 10.xxx.xxx.16 global public
2013-12-31 15:05:51: <PRIVATE_INTERFACE> 172.xx.x.16 global cluster_interconnect
....
....
2013-12-31 15:05:51: ---resulting upgrade iflist:
2013-12-31 15:05:51: intf 0: -<PUBLIC_INTERFACE>-10.xxx.xxx.16-global-public--
2013-12-31 15:05:51: intf 1: -<PRIVATE_INTERFACE>172.xx.x.16-global-cluster_interconnect--
2013-12-31 15:05:51: ---
2013-12-31 15:05:51: upgrade netlst: "<PUBLIC_INTERFACE>/10.xxx.xxx.16:public,<PRIVATE_INTERFACE>/172.xx.x.16:cluster_interconnect"
2013-12-31 15:05:51: upgrade node list: "<HOSTNAME01>,<HOSTNAME02>"
2013-12-31 15:05:51: old networks =<PUBLIC_INTERFACE>/10.xxx.xxx.16:public,<PRIVATE_INTERFACE>/172.xx.x.16:cluster_interconnect                         ====>> private network is 172.16.0.16
2013-12-31 15:05:51: old nodes =<HOSTNAME01>,<HOSTNAME02>
2013-12-31 15:05:51: old CrsHome =/<10.2.0_GI_HOME>
2013-12-31 15:05:51: old CrsVer =10 2 0 5 0
2013-12-31 15:05:51: old ClusterID=-1
2013-12-31 15:05:51: old OCRID =533885236
....
....
2013-12-31 15:07:47: ---Checking local gpnp setup...
2013-12-31 15:07:47: The setup file "/<11.2.0.3_GI_HOME>/gpnp/<HOSTNAME01>/profiles/peer/profile.xml" does not exist
2013-12-31 15:07:47: The setup file "/<11.2.0.3_GI_HOME>/gpnp/<HOSTNAME01/wallets/peer/cwallet.sso" does not exist
2013-12-31 15:07:47: The setup file "/<11.2.0.3_GI_HOME>/gpnp/<HOSTNAME01/wallets/prdr/cwallet.sso" does not exist
2013-12-31 15:07:47: chk gpnphome /<11.2.0.3_GI_HOME>/gpnp/<HOSTNAME01: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2013-12-31 15:07:47: chk gpnphome /<11.2.0.3_GI_HOME>/gpnp/<HOSTNAME01: INVALID (bad profile/wallet)
2013-12-31 15:07:47: ---Checking cluster-wide gpnp setup...
2013-12-31 15:07:47: chk gpnphome /<11.2.0.3_GI_HOME>/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
....
....
2013-12-31 15:07:48: gpnp setup checked: local valid? 0 cluster-wide valid? 1
2013-12-31 15:07:48: Taking cluster-wide setup as local
....
....
2013-12-31 15:10:21: Starting CSS in clustered mode
2013-12-31 15:10:21: Executing cmd: /<11.2.0.3_GI_HOME>/bin/crsctl start resource ora.cssd -init
2013-12-31 15:20:29: Command output:
> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sda1-db1-1-sfm'
> CRS-2672: Attempting to start 'ora.gipcd' on '<HOSTNAME01>'
> CRS-2676: Start of 'ora.cssdmonitor' on '<HOSTNAME01>' succeeded
> CRS-2676: Start of 'ora.gipcd' on '<HOSTNAME01>' succeeded
> CRS-2672: Attempting to start 'ora.cssd' on '<HOSTNAME01>-sfm'
> CRS-2676: Start of 'ora.diskmon' on '<HOSTNAME01>' succeeded
> CRS-2674: Start of 'ora.cssd' on '<HOSTNAME01>' failed
> CRS-2679: Attempting to clean 'ora.cssd' on '<HOSTNAME01>'
> CRS-2681: Clean of 'ora.cssd' on '<HOSTNAME01>' succeeded
> CRS-2673: Attempting to stop 'ora.gipcd' on '<HOSTNAME01>'
> CRS-2677: Stop of 'ora.gipcd' on '<HOSTNAME01>' succeeded
> CRS-2673: Attempting to stop 'ora.cssdmonitor' on '<HOSTNAME01>'
> CRS-2677: Stop of 'ora.cssdmonitor' on '<HOSTNAME01>' succeeded
> CRS-5804: Communication error with agent process
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
....
....
2013-12-31 15:21:21: Failed to start Oracle Grid Infrastructure stack
2013-12-31 15:21:21: ###### Begin DIE Stack Trace ######
2013-12-31 15:21:21: Package File Line Calling
2013-12-31 15:21:21: --------------- -------------------- ---- ----------
2013-12-31 15:21:21: 1: main rootcrs.pl 387 crsconfig_lib::dietrap
2013-12-31 15:21:21: 2: crsconfig_lib crsconfig_lib.pm 1238 main::__ANON__
2013-12-31 15:21:21: 3: crsconfig_lib crsconfig_lib.pm 1198 crsconfig_lib::start_cluster
2013-12-31 15:21:21: 4: main rootcrs.pl 845 crsconfig_lib::perform_start_cluster
2013-12-31 15:21:21: ####### End DIE Stack Trace #######

  

2013-12-31 15:10:24.927: [ CLSINET][3186363680] Returning NETDATA: 0 interfaces                         ====>> no private network
....
....
2013-12-31 15:10:42.762: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450478, LATS 3751626906, lastSeqNo 59450477, uniqueness 1328910683, timestamp 1388502642/3748920666
2013-12-31 15:10:42.881: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:10:43.765: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450479, LATS 3751627916, lastSeqNo 59450478, uniqueness 1328910683, timestamp 1388502643/3748921676
2013-12-31 15:10:44.768: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450480, LATS 3751628916, lastSeqNo 59450479, uniqueness 1328910683, timestamp 1388502644/3748922676
2013-12-31 15:10:45.771: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450481, LATS 3751629916, lastSeqNo 59450480, uniqueness 1328910683, timestamp 1388502645/3748923676
2013-12-31 15:10:46.774: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450482, LATS 3751630916, lastSeqNo 59450481, uniqueness 1328910683, timestamp 1388502646/3748924676
2013-12-31 15:10:47.777: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450483, LATS 3751631926, lastSeqNo 59450482, uniqueness 1328910683, timestamp 1388502647/3748925686
2013-12-31 15:10:48.780: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450484, LATS 3751632926, lastSeqNo 59450483, uniqueness 1328910683, timestamp 1388502648/3748926686
....
....
2013-12-31 15:20:18.037: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:20:18.395: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02> has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451052, LATS 3752202476, lastSeqNo 59451051, uniqueness 1328910683, timestamp 1388503217/3749495976
2013-12-31 15:20:19.398: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451053, LATS 3752203476, lastSeqNo 59451052, uniqueness 1328910683, timestamp 1388503218/3749496976
2013-12-31 15:20:20.401: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451054, LATS 3752204486, lastSeqNo 59451053, uniqueness 1328910683, timestamp 1388503219/3749497986
2013-12-31 15:20:21.404: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451055, LATS 3752205486, lastSeqNo 59451054, uniqueness 1328910683, timestamp 1388503220/3749498986
2013-12-31 15:20:22.407: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, <HOSTNAME02>, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451056, LATS 3752206486, lastSeqNo 59451055, uniqueness 1328910683, timestamp 1388503221/3749499986
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x144a6310)
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmShutDown: Received abortive shutdown request from client.
2013-12-31 15:20:22.816: [ CSSD][1077659968]###################################
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssscExit: CSSD aborting from thread GMClientListener

   

2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: usterName="crs" PALocation=""><gpnp:Network-Profile><gpnp:HostNe[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: twork id="gen" HostName="*"><gpnp:Network id="net1" IP="10.224.1[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 94.0" Adapter="bond0" Use="public"/><gpnp:Network id="net2" IP="[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 172.xx.x.0" Adapter="bond1" Use="cluster_interconnect"/></gpnp:H[cont]                     ====>> wrong private network
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: ostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" Dis[cont]

  



Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.