Node eviction after applied Jul 2016 GI PSU
(Doc ID 2231676.1)
Last updated on FEBRUARY 11, 2019
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
After applied Jul 2016 GI PSU, node eviction happens:
GI alert.log from surviving node
[OCSSD(5372)]CRS-1607: Nodescj5db01z2 is being evicted in cluster incarnation 373918374; details at (:CSSNM00007:) in /u01/app/oracle/diag/crs/<hostname>/crs/trace/ocssd.trc.
ocssd.trc from evicted node
2016-11-14 19:56:45.519377 : CSSD:13: clssgmcClientDestroy: (10aa27dc0) clientID 4:29008:1 cleaned up con (0), joinstate 2
====>>>> gap in trace file for more than 30 seconds before eviction
2016-11-14 19:58:40.288985 : CSSD:61: clsde_evtdata_init: evtdata [10ab16490], subtype [NETWORK], mod [CSSD], time [2016-11-14 18:58:40.288], loc [<hostname>], comment [50% network heartbeat failed for node <hostname>]
2016-11-14 19:58:40.289103 : CSSD:61: clsde_clsceevt_publish: Clusterwide event
2016-11-14 19:58:40.294803 : CSSD:61: clsde_evtdata_destroy: pdata [10ab16490], subtype [NETWORK] mod [CSSD], time [2016-11-14 18:58:40.288], loc [scj5db02z1], comment [50% network heartbeat failed for node <hostname>]
====>>>> gap in trace file for more than 30 seconds before eviction
2016-11-14 19:58:40.288985 : CSSD:61: clsde_evtdata_init: evtdata [10ab16490], subtype [NETWORK], mod [CSSD], time [2016-11-14 18:58:40.288], loc [<hostname>], comment [50% network heartbeat failed for node <hostname>]
2016-11-14 19:58:40.289103 : CSSD:61: clsde_clsceevt_publish: Clusterwide event
2016-11-14 19:58:40.294803 : CSSD:61: clsde_evtdata_destroy: pdata [10ab16490], subtype [NETWORK] mod [CSSD], time [2016-11-14 18:58:40.288], loc [scj5db02z1], comment [50% network heartbeat failed for node <hostname>]
pstack of ocssd.bin from evicted node
lwp_park
clsdadr_bucket_syncb ====>>>> clsdadr_bucket_syncb on the top
dbgtrReserveSpace
....
....
gipcWaitF
clssscSelect
clssnmClusterListener
clsdadr_bucket_syncb ====>>>> clsdadr_bucket_syncb on the top
dbgtrReserveSpace
....
....
gipcWaitF
clssscSelect
clssnmClusterListener
Changes
Applied Jul 2016 GI PSU.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
References |