CSSD Fails to Join the Cluster After Private Network Recovered if avahi Daemon is up and Running
(Doc ID 1501093.1)
Last updated on NOVEMBER 26, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and laterOracle Database Cloud Schema Service - Version N/A and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
11gR2 Grid Infrastructure node evicted as expected after private network issue, however, after network is restored, ora.cssd on the evicted node fails to join the cluster:
- <GI_HOME>/log/<node>/alert<node>.log - surviving node
2012-10-06 22:53:13.224
[cssd(30221)]CRS-1612:Network communication with node <hostname> (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.020 seconds
2012-10-06 22:53:20.238
[cssd(30221)]CRS-1611:Network communication with node <hostname> (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.010 seconds
2012-10-06 22:53:24.246
[cssd(30221)]CRS-1610:Network communication with node <hostname> (2) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2012-10-06 22:53:27.255
[cssd(30221)]CRS-1607:Node <hostname> is being evicted in cluster incarnation 239306494; details at (:CSSNM00007:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 22:53:30.369
<GI_HOME>/bin/oraagent.bin(17232)]CRS-5818:Aborted command 'check' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:11:2} in <GI_HOME>/log/<hostname>/agent/ohasd/oraagent_grid/oraagent_grid.log.
2012-10-06 22:53:57.876
[cssd(30221)]CRS-1601:CSSD Reconfiguration complete. Active nodes are <hostname> .
2012-10-06 22:53:58.471
[ohasd(25845)]CRS-8011:reboot advisory message from host: <hostname>, component: cssagent, with time stamp: L-2012-10-06-22:54:53.717
[ohasd(25845)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
..
[crsd(13213)]CRS-5504:Node down event reported for node '<hostname>'.
..
[crsd(13213)]CRS-2773:Server '<hostname>' has been removed from pool 'Generic'.
[cssd(30221)]CRS-1612:Network communication with node <hostname> (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.020 seconds
2012-10-06 22:53:20.238
[cssd(30221)]CRS-1611:Network communication with node <hostname> (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.010 seconds
2012-10-06 22:53:24.246
[cssd(30221)]CRS-1610:Network communication with node <hostname> (2) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2012-10-06 22:53:27.255
[cssd(30221)]CRS-1607:Node <hostname> is being evicted in cluster incarnation 239306494; details at (:CSSNM00007:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 22:53:30.369
<GI_HOME>/bin/oraagent.bin(17232)]CRS-5818:Aborted command 'check' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:11:2} in <GI_HOME>/log/<hostname>/agent/ohasd/oraagent_grid/oraagent_grid.log.
2012-10-06 22:53:57.876
[cssd(30221)]CRS-1601:CSSD Reconfiguration complete. Active nodes are <hostname> .
2012-10-06 22:53:58.471
[ohasd(25845)]CRS-8011:reboot advisory message from host: <hostname>, component: cssagent, with time stamp: L-2012-10-06-22:54:53.717
[ohasd(25845)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
..
[crsd(13213)]CRS-5504:Node down event reported for node '<hostname>'.
..
[crsd(13213)]CRS-2773:Server '<hostname>' has been removed from pool 'Generic'.
- <GI_HOME>/log/<node>/alert<node>.log - evicted node
2012-10-06 22:54:38.644
[cssd(16774)]CRS-1612:Network communication with node <hostname> (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.730 seconds
2012-10-06 22:54:46.664
[cssd(16774)]CRS-1611:Network communication with node <hostname> (1) missing for 75% of timeout interval. Removal of this node from cluster in 6.710 seconds
2012-10-06 22:54:50.674
[cssd(16774)]CRS-1610:Network communication with node <hostname> (1) missing for 90% of timeout interval. Removal of this node from cluster in 2.700 seconds
2012-10-06 22:54:53.129
[cssd(16774)]CRS-1608:This node was evicted by node 1, <hostname>; details at (:CSSNM00005:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 22:54:53.130
[cssd(16774)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log
2012-10-06 22:54:53.130
[cssd(16774)]CRS-1652:Starting clean up of CRSD resources.
2012-10-06 22:54:53.320
[cssd(16774)]CRS-1653:The clean up of the CRSD resources failed.
2012-10-06 23:08:47.767
[ohasd(26510)]CRS-2112:The OLR service started on node <hostname>.
2012-10-06 23:08:47.923
[ohasd(26510)]CRS-1301:Oracle High Availability Service started on node <hostname>.
2012-10-06 23:08:48.013
[ohasd(26510)]CRS-8011:reboot advisory message from host: <hostname>, component: cssagent, with time stamp: L-2012-10-06-22:54:53.717
[ohasd(26510)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
2012-10-06 23:08:48.120
[ohasd(26510)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 1 were announced and 0 errors occurred
..
[ohasd(26510)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-10-06 23:08:54.297
[gpnpd(26944)]CRS-2328:GPNPD started on node <hostname>.
2012-10-06 23:08:56.836
[cssd(27059)]CRS-1713:CSSD daemon is started in clustered mode
2012-10-06 23:08:58.572
[ohasd(26510)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2012-10-06 23:09:06.760
[cssd(27059)]CRS-1707:Lease acquisition for node <hostname> number 2 completed
2012-10-06 23:09:08.063
[cssd(27059)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/OCRDATA1; details in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 23:18:55.845
[/u02/grid/app/11.2.0/grid/bin/cssdagent(27047)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in <GI_HOME>/log/<hostname>/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-10-06 23:18:55.846
[cssd(27059)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log
##>> CSSD fails to join even though network is fine
[cssd(16774)]CRS-1612:Network communication with node <hostname> (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.730 seconds
2012-10-06 22:54:46.664
[cssd(16774)]CRS-1611:Network communication with node <hostname> (1) missing for 75% of timeout interval. Removal of this node from cluster in 6.710 seconds
2012-10-06 22:54:50.674
[cssd(16774)]CRS-1610:Network communication with node <hostname> (1) missing for 90% of timeout interval. Removal of this node from cluster in 2.700 seconds
2012-10-06 22:54:53.129
[cssd(16774)]CRS-1608:This node was evicted by node 1, <hostname>; details at (:CSSNM00005:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 22:54:53.130
[cssd(16774)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log
2012-10-06 22:54:53.130
[cssd(16774)]CRS-1652:Starting clean up of CRSD resources.
2012-10-06 22:54:53.320
[cssd(16774)]CRS-1653:The clean up of the CRSD resources failed.
2012-10-06 23:08:47.767
[ohasd(26510)]CRS-2112:The OLR service started on node <hostname>.
2012-10-06 23:08:47.923
[ohasd(26510)]CRS-1301:Oracle High Availability Service started on node <hostname>.
2012-10-06 23:08:48.013
[ohasd(26510)]CRS-8011:reboot advisory message from host: <hostname>, component: cssagent, with time stamp: L-2012-10-06-22:54:53.717
[ohasd(26510)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
2012-10-06 23:08:48.120
[ohasd(26510)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 1 were announced and 0 errors occurred
..
[ohasd(26510)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-10-06 23:08:54.297
[gpnpd(26944)]CRS-2328:GPNPD started on node <hostname>.
2012-10-06 23:08:56.836
[cssd(27059)]CRS-1713:CSSD daemon is started in clustered mode
2012-10-06 23:08:58.572
[ohasd(26510)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2012-10-06 23:09:06.760
[cssd(27059)]CRS-1707:Lease acquisition for node <hostname> number 2 completed
2012-10-06 23:09:08.063
[cssd(27059)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/OCRDATA1; details in <GI_HOME>/log/<hostname>/cssd/ocssd.log.
2012-10-06 23:18:55.845
[/u02/grid/app/11.2.0/grid/bin/cssdagent(27047)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in <GI_HOME>/log/<hostname>/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-10-06 23:18:55.846
[cssd(27059)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <GI_HOME>/log/<hostname>/cssd/ocssd.log
##>> CSSD fails to join even though network is fine
2012-10-06 23:18:55.847
[cssd(27059)]CRS-1603:CSSD on node <hostname> shutdown by user.
2012-10-06 23:19:00.969
[ohasd(26510)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server '<hostname>'.
2012-10-06 23:19:01.105
[ohasd(26510)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2012-10-06 23:19:01.105
[ohasd(26510)]CRS-2807:Resource 'ora.cluster_interconnect.haip' failed to start automatically.
2012-10-06 23:19:01.105
[ohasd(26510)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2012-10-06 23:19:01.105
[ohasd(26510)]CRS-2807:Resource 'ora.cssd' failed to start automatically.
2012-10-06 23:19:01.105
[ohasd(26510)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2012-10-06 23:19:01.106
[ohasd(26510)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
- <GI_HOME>/log/<node>/cssd/ocssd.log - surviving node
2012-10-06 22:52:57.098: [ CSSD][1111382336]clssgmFreeRPCIndex: freeing rpc 1054
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: [network] failed send attempt endp 0x2aaab4210c10 [0000000004a1b64b] { gipcEndpoint : localAddr 'udp://192.xx.xx.6:45968', remoteAddr '', numPend 5, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x2, usrFlags 0x4000 }, req 0x2aaab41218e0 [0000000004a23db4] { gipcSendRequest : addr 'udp://192.xx.xx.7:59525', data 0x2aaab4139c38, len 228, olen 0, parentEndp 0x2aaab4210c10, ret gipcretEndpointNotAvailable (40), objFlags 0x0, reqFlags 0x2 }
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos op : sgipcnValidateSocket
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos dep : Invalid argument (22)
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos loc : address not
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos info: addr '192.xx.xx.6:45968', len 228, buf 0x2aaab4139c38, cookie 0x2aaab41218e0
2012-10-06 22:52:58.801: [GIPCXCPT][1084971328] gipcInternalSendSync: failed sync request, ret gipcretEndpointNotAvailable (40)
2012-10-06 22:52:58.801: [GIPCXCPT][1084971328] gipcSendSyncF [gipchaLowerInternalSend : gipchaLower.c : 793]: EXCEPTION[ ret gipcretEndpointNotAvailable (40) ] failed to send on endp 0x2aaab4210c10 [0000000004a1b64b] { gipcEndpoint : localAddr 'udp://192.xx.xx.6:45968', remoteAddr '', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x2, usrFlags 0x4000 }, addr 0x2aaaac598aa0 [0000000004a1f1c0] { gipcAddress : name 'udp://192.xx.xx.7:59525', objFlags 0x0, addrFlags 0x1 }, buf 0x2aaab4139c38, len 228, flags 0x0
2012-10-06 22:52:58.802: [GIPCHGEN][1084971328] gipchaInterfaceFail: marking interface failing 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x6 }
2012-10-06 22:52:58.802: [GIPCHALO][1084971328] gipchaLowerInternalSend: failed to initiate send on interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x86 }, hctx 0x17cd8270 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_GESBIDB-cluster', luid '17660728-00000000', numNode 2, numInf 1, usrFlags 0x0, flags 0x67 }
2012-10-06 22:52:59.227: [GIPCHGEN][1084971328] gipchaInterfaceFail: marking interface failing 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x6 }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab80b8180 { host '', haName 'CSS_GESBIDB-cluster', local (nil), ip '192.xx.xx.6:45968', subnet '192.xx.xx.0', mask '255.255.255.0', mac 'a0-b3-cc-1c-8b-68', ifname 'bond1', numRef 0, numFail 2, idxBoot 0, flags 0x194d }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x86 }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x86 }
2012-10-06 22:52:59.804: [GIPCHALO][1084971328] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0xa6 }
2012-10-06 22:53:00.180: [ CSSD][1102190912]clssgmDeadProc: proc 0x17fdffc0
..
2012-10-06 22:53:00.805: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:00.805: [ CSSD][1114536256]clssnmSendingThread: sent 5 status msgs to all nodes
2012-10-06 22:53:00.805: [GIPCHGEN][1084971328] gipchaNodeAddInterface: adding interface information for inf 0x2aaab4215260 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local (nil), ip '192.xx.xx.6:45968', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x1842 }
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'mcast://224.0.0.251:42424/192.xx.xx.6:45968'
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'mcast://230.0.1.0:42424/192.xx.xx.6:45968'
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap broadcast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'udp://192.xx.xx.255:45968'
2012-10-06 22:53:00.806: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:01.295: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:01.807: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:02.808: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
..
2012-10-06 22:53:32.263: [ CSSD][1081456960]clssnmvDiskKillCheck: not evicted, file /dev/oracleasm/disks/OCRDATA1 flags 0x00000000, kill block unique 0, my unique 1348253376
2012-10-06 22:53:32.268: [ CSSD][1089239360]clssnmvDiskPing: Writing with status 0x3, timestamp 649255908/1349544212
2012-10-06 22:53:32.288: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:32.389: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
..
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEviction: Node kill could not beperformed. Admin or connection validation failed
2012-10-06 22:53:54.926: [ CSSD][1089239360]clssnmvDiskPing: Writing with status 0x3, timestamp 649278558/1349544234
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEviction: Node kill could not beperformed. Admin or connection validation failed
..
2012-10-06 22:53:55.410: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:55.410: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
..
2012-10-06 22:53:57.855: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:57.855: [ CSSD][1116113216]clssnmCheckKillStatus: Node 2, <hostname>, down, LATS(649251398),timeout(30090)
..
2012-10-06 23:02:53.660: [GIPCHALO][1084971328] gipchaLowerProcessNode: no valid interfaces found to node for 594770 ms, node 0x2aaab80a7590 { host '<hostname>', haName 'CSS_GESBIDB-cluster', srcLuid 17660728-09a15715, dstLuid cefb875b-54a7a279 numInf 1, contigSeq 624, lastAck 675, lastValidAck 624, sendSeq [1042 : 1042], createTime 648945618, sentRegister 1, localMonitor 1, flags 0x2008 }
2012-10-06 23:02:54.055: [ CSSD][1114536256]clssnmSendingThread: sending status msg to all nodes
2012-10-06 23:02:54.056: [ CSSD][1114536256]clssnmSendingThread: sent 5 status msgs to all nodes
2012-10-06 23:02:58.660: [GIPCHALO][1084971328] gipchaLowerProcessNode: bootstrap node considered dead because of idle connection time 600770 ms, node 0x2aaab80a7590 { host '<hostname>', haName 'CSS_GESBIDB-cluster', srcLuid 17660728-09a15715, dstLuid cefb875b-54a7a279 numInf 1, contigSeq 624, lastAck 675, lastValidAck 624, sendSeq [1044 : 1044], createTime 648945618, sentRegister 1, localMonitor 1, flags 0x2408 }
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: [network] failed send attempt endp 0x2aaab4210c10 [0000000004a1b64b] { gipcEndpoint : localAddr 'udp://192.xx.xx.6:45968', remoteAddr '', numPend 5, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x2, usrFlags 0x4000 }, req 0x2aaab41218e0 [0000000004a23db4] { gipcSendRequest : addr 'udp://192.xx.xx.7:59525', data 0x2aaab4139c38, len 228, olen 0, parentEndp 0x2aaab4210c10, ret gipcretEndpointNotAvailable (40), objFlags 0x0, reqFlags 0x2 }
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos op : sgipcnValidateSocket
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos dep : Invalid argument (22)
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos loc : address not
2012-10-06 22:52:58.801: [ GIPCNET][1084971328] gipcmodNetworkProcessSend: slos info: addr '192.xx.xx.6:45968', len 228, buf 0x2aaab4139c38, cookie 0x2aaab41218e0
2012-10-06 22:52:58.801: [GIPCXCPT][1084971328] gipcInternalSendSync: failed sync request, ret gipcretEndpointNotAvailable (40)
2012-10-06 22:52:58.801: [GIPCXCPT][1084971328] gipcSendSyncF [gipchaLowerInternalSend : gipchaLower.c : 793]: EXCEPTION[ ret gipcretEndpointNotAvailable (40) ] failed to send on endp 0x2aaab4210c10 [0000000004a1b64b] { gipcEndpoint : localAddr 'udp://192.xx.xx.6:45968', remoteAddr '', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x2, usrFlags 0x4000 }, addr 0x2aaaac598aa0 [0000000004a1f1c0] { gipcAddress : name 'udp://192.xx.xx.7:59525', objFlags 0x0, addrFlags 0x1 }, buf 0x2aaab4139c38, len 228, flags 0x0
2012-10-06 22:52:58.802: [GIPCHGEN][1084971328] gipchaInterfaceFail: marking interface failing 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x6 }
2012-10-06 22:52:58.802: [GIPCHALO][1084971328] gipchaLowerInternalSend: failed to initiate send on interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x86 }, hctx 0x17cd8270 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_GESBIDB-cluster', luid '17660728-00000000', numNode 2, numInf 1, usrFlags 0x0, flags 0x67 }
2012-10-06 22:52:59.227: [GIPCHGEN][1084971328] gipchaInterfaceFail: marking interface failing 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x6 }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab80b8180 { host '', haName 'CSS_GESBIDB-cluster', local (nil), ip '192.xx.xx.6:45968', subnet '192.xx.xx.0', mask '255.255.255.0', mac 'a0-b3-cc-1c-8b-68', ifname 'bond1', numRef 0, numFail 2, idxBoot 0, flags 0x194d }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0x86 }
2012-10-06 22:52:59.803: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x86 }
2012-10-06 22:52:59.804: [GIPCHALO][1084971328] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x2aaaac9bbd50 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:59525', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 1, flags 0xa6 }
2012-10-06 22:53:00.180: [ CSSD][1102190912]clssgmDeadProc: proc 0x17fdffc0
..
2012-10-06 22:53:00.805: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:00.805: [ CSSD][1114536256]clssnmSendingThread: sent 5 status msgs to all nodes
2012-10-06 22:53:00.805: [GIPCHGEN][1084971328] gipchaNodeAddInterface: adding interface information for inf 0x2aaab4215260 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local (nil), ip '192.xx.xx.6:45968', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0x1842 }
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'mcast://224.0.0.251:42424/192.xx.xx.6:45968'
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'mcast://230.0.1.0:42424/192.xx.xx.6:45968'
2012-10-06 22:53:00.805: [GIPCHTHR][1084971328] gipchaWorkerCreateInterface: created remote bootstrap broadcast interface for node '<hostname>', haName 'CSS_GESBIDB-cluster', inf 'udp://192.xx.xx.255:45968'
2012-10-06 22:53:00.806: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:01.295: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:01.807: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
2012-10-06 22:53:02.808: [GIPCHGEN][1084971328] gipchaInterfaceDisable: disabling interface 0x2aaab0155970 { host '<hostname>', haName 'CSS_GESBIDB-cluster', local 0x2aaab80b8180, ip '192.xx.xx.7:46712', subnet '192.xx.xx.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 0, flags 0xa6 }
..
2012-10-06 22:53:32.263: [ CSSD][1081456960]clssnmvDiskKillCheck: not evicted, file /dev/oracleasm/disks/OCRDATA1 flags 0x00000000, kill block unique 0, my unique 1348253376
2012-10-06 22:53:32.268: [ CSSD][1089239360]clssnmvDiskPing: Writing with status 0x3, timestamp 649255908/1349544212
2012-10-06 22:53:32.288: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:32.389: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
..
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
2012-10-06 22:53:54.834: [ CSSD][1116113216]clssnmWaitOnEviction: Node kill could not beperformed. Admin or connection validation failed
2012-10-06 22:53:54.926: [ CSSD][1089239360]clssnmvDiskPing: Writing with status 0x3, timestamp 649278558/1349544234
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
2012-10-06 22:53:54.929: [ CSSD][1116113216]clssnmWaitOnEviction: Node kill could not beperformed. Admin or connection validation failed
..
2012-10-06 22:53:55.410: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:55.410: [ CSSD][1116113216]clssnmWaitOnEviction: node(2) exceeded graceful shutdown period, IPMI-kill allowed if configured
..
2012-10-06 22:53:57.855: [ CSSD][1116113216]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1349544292, 643367598, 309), seedhbimpd TRUE
2012-10-06 22:53:57.855: [ CSSD][1116113216]clssnmCheckKillStatus: Node 2, <hostname>, down, LATS(649251398),timeout(30090)
..
2012-10-06 23:02:53.660: [GIPCHALO][1084971328] gipchaLowerProcessNode: no valid interfaces found to node for 594770 ms, node 0x2aaab80a7590 { host '<hostname>', haName 'CSS_GESBIDB-cluster', srcLuid 17660728-09a15715, dstLuid cefb875b-54a7a279 numInf 1, contigSeq 624, lastAck 675, lastValidAck 624, sendSeq [1042 : 1042], createTime 648945618, sentRegister 1, localMonitor 1, flags 0x2008 }
2012-10-06 23:02:54.055: [ CSSD][1114536256]clssnmSendingThread: sending status msg to all nodes
2012-10-06 23:02:54.056: [ CSSD][1114536256]clssnmSendingThread: sent 5 status msgs to all nodes
2012-10-06 23:02:58.660: [GIPCHALO][1084971328] gipchaLowerProcessNode: bootstrap node considered dead because of idle connection time 600770 ms, node 0x2aaab80a7590 { host '<hostname>', haName 'CSS_GESBIDB-cluster', srcLuid 17660728-09a15715, dstLuid cefb875b-54a7a279 numInf 1, contigSeq 624, lastAck 675, lastValidAck 624, sendSeq [1044 : 1044], createTime 648945618, sentRegister 1, localMonitor 1, flags 0x2408 }
- <GI_HOME>/log/<node>/cssd/ocssd.log - evicted node
2012-10-06 22:54:38.644: [ CSSD][1111112000]clssnmPollingThread: node <hostname> (1) at 50% heartbeat fatal, removal in 14.730 seconds
2012-10-06 22:54:38.644: [ CSSD][1111112000]clssnmPollingThread: node <hostname> (1) is impending reconfig, flag 2491406, misstime 15270
2012-10-06 22:54:38.644: [ CSSD][1111112000]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
..
2012-10-06 22:54:52.180: [ CSSD][1079523648]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306494, wrtcnt, 5677604, LATS 643366798, lastSeqNo 5677603, uniqueness 1348253376, timestamp 1349544206/649249798
2012-10-06 22:54:52.469: [ CSSD][1103849792]clssnmvDiskPing: Writing with status 0x3, timestamp 643367078/1349544292
2012-10-06 22:54:52.679: [ CSSD][1079523648]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306494, wrtcnt, 5677605, LATS 643367288, lastSeqNo 5677604, uniqueness 1348253376, timestamp 1349544206/649250308
2012-10-06 22:54:52.981: [ CSSD][1103849792]clssnmvDiskPing: Writing with status 0x3, timestamp 643367598/1349544292
2012-10-06 22:54:53.130: [ CSSD][1092438336](:CSSNM00005:)clssnmvDiskKillCheck: Aborting, evicted by node <hostname>, number 1, sync 239306494, stamp 649250888
2012-10-06 22:54:53.130: [ CSSD][1092438336]###################################
2012-10-06 22:54:53.130: [ CSSD][1092438336]clssscExit: CSSD aborting from thread clssnmvKillBlockThread
2012-10-06 22:54:53.130: [ CSSD][1092438336]###################################
2012-10-06 22:54:53.130: [ CSSD][1092438336](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
..
2012-10-06 23:08:56.835: [ CSSD][4037557536]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value 1349545136
..
2012-10-06 23:18:55.844: [ CSSD][1080105280]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306495, wrtcnt, 5678923, LATS 37504, lastSeqNo 5678922, uniqueness 1348253376, timestamp 1349545498/650541868
2012-10-06 23:18:55.846: [ CSSD][1101527360]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x10a21430)
##>> CSSD fails to join even though network is fine
2012-10-06 22:54:38.644: [ CSSD][1111112000]clssnmPollingThread: node <hostname> (1) is impending reconfig, flag 2491406, misstime 15270
2012-10-06 22:54:38.644: [ CSSD][1111112000]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
..
2012-10-06 22:54:52.180: [ CSSD][1079523648]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306494, wrtcnt, 5677604, LATS 643366798, lastSeqNo 5677603, uniqueness 1348253376, timestamp 1349544206/649249798
2012-10-06 22:54:52.469: [ CSSD][1103849792]clssnmvDiskPing: Writing with status 0x3, timestamp 643367078/1349544292
2012-10-06 22:54:52.679: [ CSSD][1079523648]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306494, wrtcnt, 5677605, LATS 643367288, lastSeqNo 5677604, uniqueness 1348253376, timestamp 1349544206/649250308
2012-10-06 22:54:52.981: [ CSSD][1103849792]clssnmvDiskPing: Writing with status 0x3, timestamp 643367598/1349544292
2012-10-06 22:54:53.130: [ CSSD][1092438336](:CSSNM00005:)clssnmvDiskKillCheck: Aborting, evicted by node <hostname>, number 1, sync 239306494, stamp 649250888
2012-10-06 22:54:53.130: [ CSSD][1092438336]###################################
2012-10-06 22:54:53.130: [ CSSD][1092438336]clssscExit: CSSD aborting from thread clssnmvKillBlockThread
2012-10-06 22:54:53.130: [ CSSD][1092438336]###################################
2012-10-06 22:54:53.130: [ CSSD][1092438336](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
..
2012-10-06 23:08:56.835: [ CSSD][4037557536]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value 1349545136
..
2012-10-06 23:18:55.844: [ CSSD][1080105280]clssnmvDHBValidateNcopy: node 1, <hostname>, has a disk HB, but no network HB, DHB has rcfg 239306495, wrtcnt, 5678923, LATS 37504, lastSeqNo 5678922, uniqueness 1348253376, timestamp 1349545498/650541868
2012-10-06 23:18:55.846: [ CSSD][1101527360]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x10a21430)
##>> CSSD fails to join even though network is fine
2012-10-06 23:18:55.846: [ CSSD][1101527360]clssgmShutDown: Received abortive shutdown request from client.
2012-10-06 23:18:55.846: [ CSSD][1101527360]###################################
2012-10-06 23:18:55.846: [ CSSD][1101527360]clssscExit: CSSD aborting from thread GMClientListener
- OS /var/log/messages - node which had network issue
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: Removing slave eth8
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: Warning: the permanent HWaddr of eth8 - A0:B3:CC:1C:8B:68 - is still in use by bond1. Set the HWaddr of eth8 to a different address to avoid conflicts.
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: releasing active interface eth8
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: making interface eth9 the new active one.
Oct 6 22:52:58 racnode1 kernel: bonding: bond1: Removing slave eth9
Oct 6 22:52:58 racnode1 kernel: bonding: bond1: releasing active interface eth9
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for 192.xx.xx.6 on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Leaving mDNS multicast group on interface bond1.IPv4 with address 192.xx.xx.6.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv4 with address 169.254.xx.xx.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for 169.254.xx.xx on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Interface bond1.IPv4 no longer relevant for mDNS.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for fe80::a2b3:ccff:fe1c:8b68 on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Leaving mDNS multicast group on interface bond1.IPv6 with address fe80::a2b3:ccff:fe1c:8b68.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Interface bond1.IPv6 no longer relevant for mDNS.
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: New relevant interface bond1.IPv4 for mDNS.
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv4 with address 169.254.xx.xx.
Oct 6 22:53:05 racnode1 kernel: ADDRCONF(NETDEV_UP): bond1: link is not ready
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: Registering new address record for 169.254.xx.xx on bond1.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: Adding slave eth8.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: making interface eth8 the new active one.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: first active interface up!
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: enslaving eth8 as an active interface with an up link.
Oct 6 22:53:05 racnode1 kernel: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: Adding slave eth9.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: enslaving eth9 as an active interface with an up link.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: New relevant interface bond1.IPv6 for mDNS.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv6 with address fe80::a2b3:ccff:fe1c:8b68.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: Registering new address record for fe80::a2b3:ccff:fe1c:8b68 on bond1.
Oct 6 22:53:09 racnode1 avahi-daemon[22477]: Registering new address record for 192.xx.xx.6 on bond1.
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: Warning: the permanent HWaddr of eth8 - A0:B3:CC:1C:8B:68 - is still in use by bond1. Set the HWaddr of eth8 to a different address to avoid conflicts.
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: releasing active interface eth8
Oct 6 22:52:57 racnode1 kernel: bonding: bond1: making interface eth9 the new active one.
Oct 6 22:52:58 racnode1 kernel: bonding: bond1: Removing slave eth9
Oct 6 22:52:58 racnode1 kernel: bonding: bond1: releasing active interface eth9
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for 192.xx.xx.6 on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Leaving mDNS multicast group on interface bond1.IPv4 with address 192.xx.xx.6.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv4 with address 169.254.xx.xx.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for 169.254.xx.xx on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Interface bond1.IPv4 no longer relevant for mDNS.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Withdrawing address record for fe80::a2b3:ccff:fe1c:8b68 on bond1.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Leaving mDNS multicast group on interface bond1.IPv6 with address fe80::a2b3:ccff:fe1c:8b68.
Oct 6 22:52:58 racnode1 avahi-daemon[22477]: Interface bond1.IPv6 no longer relevant for mDNS.
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: New relevant interface bond1.IPv4 for mDNS.
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv4 with address 169.254.xx.xx.
Oct 6 22:53:05 racnode1 kernel: ADDRCONF(NETDEV_UP): bond1: link is not ready
Oct 6 22:53:05 racnode1 avahi-daemon[22477]: Registering new address record for 169.254.xx.xx on bond1.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: Adding slave eth8.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: making interface eth8 the new active one.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: first active interface up!
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: enslaving eth8 as an active interface with an up link.
Oct 6 22:53:05 racnode1 kernel: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: Adding slave eth9.
Oct 6 22:53:05 racnode1 kernel: bonding: bond1: enslaving eth9 as an active interface with an up link.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: New relevant interface bond1.IPv6 for mDNS.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: Joining mDNS multicast group on interface bond1.IPv6 with address fe80::a2b3:ccff:fe1c:8b68.
Oct 6 22:53:07 racnode1 avahi-daemon[22477]: Registering new address record for fe80::a2b3:ccff:fe1c:8b68 on bond1.
Oct 6 22:53:09 racnode1 avahi-daemon[22477]: Registering new address record for 192.xx.xx.6 on bond1.
- avahi deamon can also spin or accumulate high CPU time
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |