My Oracle Support Banner

rootupgrade.sh Fails as CSSD Does not Start in Cloned Environment (Doc ID 1611318.1)

Last updated on JULY 26, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading from 10.2.0.5 CRS to a cloned 11.2.0.3 GI, rootupgrade.sh fails on node1

2013-12-31 15:05:40: The active version of the Oracle Clustereware is '10 2 0 5 0'
2013-12-31 15:05:40: Skipping clusterguid fetch for 10.2.0.5.0
2013-12-31 15:05:40: Executing ocrcheck to get ocrid
2013-12-31 15:05:51: Executing cmd: /oracle/grid/11.2.0.3/bin/oifcfg iflist -p -n
2013-12-31 15:05:51: Command output:
> bond0 10.224.116.16 PRIVATE 255.255.255.240
> bond1 172.16.0.16 PRIVATE 255.255.255.240
....
....
2013-12-31 15:05:51: Earlier version Oracle Grid Infrastructure is running
2013-12-31 15:05:51: Executing cmd: /oracle/crs/product/10.2.0/bin/oifcfg getif
2013-12-31 15:05:51: Command output:
> bond0 10.224.116.16 global public
> bond1 172.16.0.16 global cluster_interconnect
>End Command output
2013-12-31 15:05:51: ---Got oifcfg out (/oracle/crs/product/10.2.0/bin/oifcfg getif):
2013-12-31 15:05:51: bond0 10.224.116.16 global public
2013-12-31 15:05:51: bond1 172.16.0.16 global cluster_interconnect
....
....
2013-12-31 15:05:51: ---resulting upgrade iflist:
2013-12-31 15:05:51: intf 0: -bond0-10.224.116.16-global-public--
2013-12-31 15:05:51: intf 1: -bond1-172.16.0.16-global-cluster_interconnect--
2013-12-31 15:05:51: ---
2013-12-31 15:05:51: upgrade netlst: "bond0/10.224.116.16:public,bond1/172.16.0.16:cluster_interconnect"
2013-12-31 15:05:51: upgrade node list: "sda1-db1-1-sfm,sda1-db1-2-sfm"
2013-12-31 15:05:51: old networks =bond0/10.224.116.16:public,bond1/172.16.0.16:cluster_interconnect                         ====>> private network is 172.16.0.16
2013-12-31 15:05:51: old nodes =sda1-db1-1-sfm,sda1-db1-2-sfm
2013-12-31 15:05:51: old CrsHome =/oracle/crs/product/10.2.0
2013-12-31 15:05:51: old CrsVer =10 2 0 5 0
2013-12-31 15:05:51: old ClusterID=-1
2013-12-31 15:05:51: old OCRID =533885236
....
....
2013-12-31 15:07:47: ---Checking local gpnp setup...
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/profiles/peer/profile.xml" does not exist
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/wallets/peer/cwallet.sso" does not exist
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/wallets/prdr/cwallet.sso" does not exist
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm: INVALID (bad profile/wallet)
2013-12-31 15:07:47: ---Checking cluster-wide gpnp setup...
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
....
....
2013-12-31 15:07:48: gpnp setup checked: local valid? 0 cluster-wide valid? 1
2013-12-31 15:07:48: Taking cluster-wide setup as local
....
....
2013-12-31 15:10:21: Starting CSS in clustered mode
2013-12-31 15:10:21: Executing cmd: /oracle/grid/11.2.0.3/bin/crsctl start resource ora.cssd -init
2013-12-31 15:20:29: Command output:
> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sda1-db1-1-sfm'
> CRS-2672: Attempting to start 'ora.gipcd' on 'sda1-db1-1-sfm'
> CRS-2676: Start of 'ora.cssdmonitor' on 'sda1-db1-1-sfm' succeeded
> CRS-2676: Start of 'ora.gipcd' on 'sda1-db1-1-sfm' succeeded
> CRS-2672: Attempting to start 'ora.cssd' on 'sda1-db1-1-sfm'
> CRS-2672: Attempting to start 'ora.diskmon' on 'sda1-db1-1-sfm'
> CRS-2676: Start of 'ora.diskmon' on 'sda1-db1-1-sfm' succeeded
> CRS-2674: Start of 'ora.cssd' on 'sda1-db1-1-sfm' failed
> CRS-2679: Attempting to clean 'ora.cssd' on 'sda1-db1-1-sfm'
> CRS-2681: Clean of 'ora.cssd' on 'sda1-db1-1-sfm' succeeded
> CRS-2673: Attempting to stop 'ora.gipcd' on 'sda1-db1-1-sfm'
> CRS-2677: Stop of 'ora.gipcd' on 'sda1-db1-1-sfm' succeeded
> CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'sda1-db1-1-sfm'
> CRS-2677: Stop of 'ora.cssdmonitor' on 'sda1-db1-1-sfm' succeeded
> CRS-5804: Communication error with agent process
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
....
....
2013-12-31 15:21:21: Failed to start Oracle Grid Infrastructure stack
2013-12-31 15:21:21: ###### Begin DIE Stack Trace ######
2013-12-31 15:21:21: Package File Line Calling
2013-12-31 15:21:21: --------------- -------------------- ---- ----------
2013-12-31 15:21:21: 1: main rootcrs.pl 387 crsconfig_lib::dietrap
2013-12-31 15:21:21: 2: crsconfig_lib crsconfig_lib.pm 1238 main::__ANON__
2013-12-31 15:21:21: 3: crsconfig_lib crsconfig_lib.pm 1198 crsconfig_lib::start_cluster
2013-12-31 15:21:21: 4: main rootcrs.pl 845 crsconfig_lib::perform_start_cluster
2013-12-31 15:21:21: ####### End DIE Stack Trace #######

  

2013-12-31 15:10:24.927: [ CLSINET][3186363680] Returning NETDATA: 0 interfaces                         ====>> no private network
....
....
2013-12-31 15:10:42.762: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450478, LATS 3751626906, lastSeqNo 59450477, uniqueness 1328910683, timestamp 1388502642/3748920666
2013-12-31 15:10:42.881: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:10:43.765: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450479, LATS 3751627916, lastSeqNo 59450478, uniqueness 1328910683, timestamp 1388502643/3748921676
2013-12-31 15:10:44.768: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450480, LATS 3751628916, lastSeqNo 59450479, uniqueness 1328910683, timestamp 1388502644/3748922676
2013-12-31 15:10:45.771: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450481, LATS 3751629916, lastSeqNo 59450480, uniqueness 1328910683, timestamp 1388502645/3748923676
2013-12-31 15:10:46.774: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450482, LATS 3751630916, lastSeqNo 59450481, uniqueness 1328910683, timestamp 1388502646/3748924676
2013-12-31 15:10:47.777: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450483, LATS 3751631926, lastSeqNo 59450482, uniqueness 1328910683, timestamp 1388502647/3748925686
2013-12-31 15:10:48.780: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450484, LATS 3751632926, lastSeqNo 59450483, uniqueness 1328910683, timestamp 1388502648/3748926686
....
....
2013-12-31 15:20:18.037: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:20:18.395: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451052, LATS 3752202476, lastSeqNo 59451051, uniqueness 1328910683, timestamp 1388503217/3749495976
2013-12-31 15:20:19.398: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451053, LATS 3752203476, lastSeqNo 59451052, uniqueness 1328910683, timestamp 1388503218/3749496976
2013-12-31 15:20:20.401: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451054, LATS 3752204486, lastSeqNo 59451053, uniqueness 1328910683, timestamp 1388503219/3749497986
2013-12-31 15:20:21.404: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451055, LATS 3752205486, lastSeqNo 59451054, uniqueness 1328910683, timestamp 1388503220/3749498986
2013-12-31 15:20:22.407: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451056, LATS 3752206486, lastSeqNo 59451055, uniqueness 1328910683, timestamp 1388503221/3749499986
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x144a6310)
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmShutDown: Received abortive shutdown request from client.
2013-12-31 15:20:22.816: [ CSSD][1077659968]###################################
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssscExit: CSSD aborting from thread GMClientListener

   

2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: usterName="crs" PALocation=""><gpnp:Network-Profile><gpnp:HostNe[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: twork id="gen" HostName="*"><gpnp:Network id="net1" IP="10.224.1[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 94.0" Adapter="bond0" Use="public"/><gpnp:Network id="net2" IP="[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 172.16.0.0" Adapter="bond1" Use="cluster_interconnect"/></gpnp:H[cont]                     ====>> wrong private network
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: ostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" Dis[cont]

  



Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process and therefore has not been subject to an independent technical review.
My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.