rootupgrade.sh Fails as CSSD Does not Start in Cloned Environment (Doc ID 1611318.1)

Last updated on JANUARY 03, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading from 10.2.0.5 CRS to a cloned 11.2.0.3 GI, rootupgrade.sh fails on node1

2013-12-31 15:05:40: The active version of the Oracle Clustereware is '10 2 0 5 0'
2013-12-31 15:05:40: Skipping clusterguid fetch for 10.2.0.5.0
2013-12-31 15:05:40: Executing ocrcheck to get ocrid
2013-12-31 15:05:51: Executing cmd: /oracle/grid/11.2.0.3/bin/oifcfg iflist -p -n
2013-12-31 15:05:51: Command output:
> bond0 10.224.116.16 PRIVATE 255.255.255.240
> bond1 172.16.0.16 PRIVATE 255.255.255.240
....
....
2013-12-31 15:05:51: Earlier version Oracle Grid Infrastructure is running
2013-12-31 15:05:51: Executing cmd: /oracle/crs/product/10.2.0/bin/oifcfg getif
2013-12-31 15:05:51: Command output:
> bond0 10.224.116.16 global public
> bond1 172.16.0.16 global cluster_interconnect
>End Command output
2013-12-31 15:05:51: ---Got oifcfg out (/oracle/crs/product/10.2.0/bin/oifcfg getif):
2013-12-31 15:05:51: bond0 10.224.116.16 global public
2013-12-31 15:05:51: bond1 172.16.0.16 global cluster_interconnect
....
....
2013-12-31 15:05:51: ---resulting upgrade iflist:
2013-12-31 15:05:51: intf 0: -bond0-10.224.116.16-global-public--
2013-12-31 15:05:51: intf 1: -bond1-172.16.0.16-global-cluster_interconnect--
2013-12-31 15:05:51: ---
2013-12-31 15:05:51: upgrade netlst: "bond0/10.224.116.16:public,bond1/172.16.0.16:cluster_interconnect"
2013-12-31 15:05:51: upgrade node list: "sda1-db1-1-sfm,sda1-db1-2-sfm"
2013-12-31 15:05:51: old networks =bond0/10.224.116.16:public,bond1/172.16.0.16:cluster_interconnect                         ====>> private network is 172.16.0.16
2013-12-31 15:05:51: old nodes =sda1-db1-1-sfm,sda1-db1-2-sfm
2013-12-31 15:05:51: old CrsHome =/oracle/crs/product/10.2.0
2013-12-31 15:05:51: old CrsVer =10 2 0 5 0
2013-12-31 15:05:51: old ClusterID=-1
2013-12-31 15:05:51: old OCRID =533885236
....
....
2013-12-31 15:07:47: ---Checking local gpnp setup...
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/profiles/peer/profile.xml" does not exist
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/wallets/peer/cwallet.sso" does not exist
2013-12-31 15:07:47: The setup file "/oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm/wallets/prdr/cwallet.sso" does not exist
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp/sda1-db1-1-sfm: INVALID (bad profile/wallet)
2013-12-31 15:07:47: ---Checking cluster-wide gpnp setup...
2013-12-31 15:07:47: chk gpnphome /oracle/grid/11.2.0.3/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
....
....
2013-12-31 15:07:48: gpnp setup checked: local valid? 0 cluster-wide valid? 1
2013-12-31 15:07:48: Taking cluster-wide setup as local
....
....
2013-12-31 15:10:21: Starting CSS in clustered mode
2013-12-31 15:10:21: Executing cmd: /oracle/grid/11.2.0.3/bin/crsctl start resource ora.cssd -init
2013-12-31 15:20:29: Command output:
> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sda1-db1-1-sfm'
> CRS-2672: Attempting to start 'ora.gipcd' on 'sda1-db1-1-sfm'
> CRS-2676: Start of 'ora.cssdmonitor' on 'sda1-db1-1-sfm' succeeded
> CRS-2676: Start of 'ora.gipcd' on 'sda1-db1-1-sfm' succeeded
> CRS-2672: Attempting to start 'ora.cssd' on 'sda1-db1-1-sfm'
> CRS-2672: Attempting to start 'ora.diskmon' on 'sda1-db1-1-sfm'
> CRS-2676: Start of 'ora.diskmon' on 'sda1-db1-1-sfm' succeeded
> CRS-2674: Start of 'ora.cssd' on 'sda1-db1-1-sfm' failed
> CRS-2679: Attempting to clean 'ora.cssd' on 'sda1-db1-1-sfm'
> CRS-2681: Clean of 'ora.cssd' on 'sda1-db1-1-sfm' succeeded
> CRS-2673: Attempting to stop 'ora.gipcd' on 'sda1-db1-1-sfm'
> CRS-2677: Stop of 'ora.gipcd' on 'sda1-db1-1-sfm' succeeded
> CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'sda1-db1-1-sfm'
> CRS-2677: Stop of 'ora.cssdmonitor' on 'sda1-db1-1-sfm' succeeded
> CRS-5804: Communication error with agent process
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
....
....
2013-12-31 15:21:21: Failed to start Oracle Grid Infrastructure stack
2013-12-31 15:21:21: ###### Begin DIE Stack Trace ######
2013-12-31 15:21:21: Package File Line Calling
2013-12-31 15:21:21: --------------- -------------------- ---- ----------
2013-12-31 15:21:21: 1: main rootcrs.pl 387 crsconfig_lib::dietrap
2013-12-31 15:21:21: 2: crsconfig_lib crsconfig_lib.pm 1238 main::__ANON__
2013-12-31 15:21:21: 3: crsconfig_lib crsconfig_lib.pm 1198 crsconfig_lib::start_cluster
2013-12-31 15:21:21: 4: main rootcrs.pl 845 crsconfig_lib::perform_start_cluster
2013-12-31 15:21:21: ####### End DIE Stack Trace #######

  

2013-12-31 15:10:24.927: [ CLSINET][3186363680] Returning NETDATA: 0 interfaces                         ====>> no private network
....
....
2013-12-31 15:10:42.762: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450478, LATS 3751626906, lastSeqNo 59450477, uniqueness 1328910683, timestamp 1388502642/3748920666
2013-12-31 15:10:42.881: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:10:43.765: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450479, LATS 3751627916, lastSeqNo 59450478, uniqueness 1328910683, timestamp 1388502643/3748921676
2013-12-31 15:10:44.768: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450480, LATS 3751628916, lastSeqNo 59450479, uniqueness 1328910683, timestamp 1388502644/3748922676
2013-12-31 15:10:45.771: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450481, LATS 3751629916, lastSeqNo 59450480, uniqueness 1328910683, timestamp 1388502645/3748923676
2013-12-31 15:10:46.774: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450482, LATS 3751630916, lastSeqNo 59450481, uniqueness 1328910683, timestamp 1388502646/3748924676
2013-12-31 15:10:47.777: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450483, LATS 3751631926, lastSeqNo 59450482, uniqueness 1328910683, timestamp 1388502647/3748925686
2013-12-31 15:10:48.780: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59450484, LATS 3751632926, lastSeqNo 59450483, uniqueness 1328910683, timestamp 1388502648/3748926686
....
....
2013-12-31 15:20:18.037: [GIPCHDEM][1083566400] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x142b2180 [0000000000000010] { gipchaContext : host 'sda1-db1-1-sfm', name 'CSS_crs', luid 'fd21098e-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2013-12-31 15:20:18.395: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451052, LATS 3752202476, lastSeqNo 59451051, uniqueness 1328910683, timestamp 1388503217/3749495976
2013-12-31 15:20:19.398: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451053, LATS 3752203476, lastSeqNo 59451052, uniqueness 1328910683, timestamp 1388503218/3749496976
2013-12-31 15:20:20.401: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451054, LATS 3752204486, lastSeqNo 59451053, uniqueness 1328910683, timestamp 1388503219/3749497986
2013-12-31 15:20:21.404: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451055, LATS 3752205486, lastSeqNo 59451054, uniqueness 1328910683, timestamp 1388503220/3749498986
2013-12-31 15:20:22.407: [ CSSD][1108199744]clssnmvDHBValidateNcopy: node 2, sda1-db1-2-sfm, has a disk HB, but no network HB, DHB has rcfg 6, wrtcnt, 59451056, LATS 3752206486, lastSeqNo 59451055, uniqueness 1328910683, timestamp 1388503221/3749499986
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x144a6310)
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssgmShutDown: Received abortive shutdown request from client.
2013-12-31 15:20:22.816: [ CSSD][1077659968]###################################
2013-12-31 15:20:22.816: [ CSSD][1077659968]clssscExit: CSSD aborting from thread GMClientListener

   

2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: usterName="crs" PALocation=""><gpnp:Network-Profile><gpnp:HostNe[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: twork id="gen" HostName="*"><gpnp:Network id="net1" IP="10.224.1[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 94.0" Adapter="bond0" Use="public"/><gpnp:Network id="net2" IP="[cont]
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: 172.16.0.0" Adapter="bond1" Use="cluster_interconnect"/></gpnp:H[cont]                     ====>> wrong private network
2013-12-31 15:10:11.514: [ GPNP][1129765808]clsgpnpd_MainWork: ostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" Dis[cont]

  



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms