Node eviction after applied Jul 2016 GI PSU (Doc ID 2231676.1)

Last updated on FEBRUARY 11, 2019

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.

Symptoms

After applied Jul 2016 GI PSU, node eviction happens:

GI alert.log from surviving node

[OCSSD(5372)]CRS-1607: Nodescj5db01z2 is being evicted in cluster incarnation 373918374; details at (:CSSNM00007:) in /u01/app/oracle/diag/crs/<hostname>/crs/trace/ocssd.trc.

ocssd.trc from evicted node

2016-11-14 19:56:45.519377 : CSSD:13: clssgmcClientDestroy: (10aa27dc0) clientID 4:29008:1 cleaned up con (0), joinstate 2
====>>>> gap in trace file for more than 30 seconds before eviction
2016-11-14 19:58:40.288985 : CSSD:61: clsde_evtdata_init: evtdata [10ab16490], subtype [NETWORK], mod [CSSD], time [2016-11-14 18:58:40.288], loc [<hostname>], comment [50% network heartbeat failed for node <hostname>]
2016-11-14 19:58:40.289103 : CSSD:61: clsde_clsceevt_publish: Clusterwide event
2016-11-14 19:58:40.294803 : CSSD:61: clsde_evtdata_destroy: pdata [10ab16490], subtype [NETWORK] mod [CSSD], time [2016-11-14 18:58:40.288], loc [scj5db02z1], comment [50% network heartbeat failed for node <hostname>]

pstack of ocssd.bin from evicted node

lwp_park
clsdadr_bucket_syncb ====>>>> clsdadr_bucket_syncb on the top
dbgtrReserveSpace
....
....
gipcWaitF
clssscSelect
clssnmClusterListener