OLOGGERD COREDUMP with CRS-8503 [crfoci_getconnstr()+532] [Signal/Exception: 11] (Doc ID 2593419.1)

2018-11-16 03:16:04.382 [OHASD(25639)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '<nodename>'
..
2018-11-16 03:19:59.731 [CRSD(12405)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 12405
2018-11-16 03:21:06.031 [CRSD(12405)]CRS-1012: The OCR service started on node <nodename>.
2018-11-16 03:21:06.303 [CRSD(12405)]CRS-1201: CRSD started on node <nodename>.
2018-11-16 03:21:12.868 [ORAAGENT(13370)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 13370
2018-11-16 03:21:13.151 [ORAROOTAGENT(13401)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 13401
2018-11-16 03:21:42.588 [ORAAGENT(14093)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 14093
2018-11-16T03:22:15.748688-08:00

2018-11-16 03:24:17.914 [OLOGGERD(12281)]CRS-8503: Oracle Clusterware process OLOGGERD with operating system process ID 12281 experienced fatal signal or exception code 11.
2018-11-16T03:24:17.935305-08:00
Errors in file <oracle_base>/crs/<nodename>/crs/trace/ologgerd.trc (incident=9):
CRS-8503 [crfoci_getconnstr()+532] [Signal/Exception: 11] [Instruction Addr: fffffffece392e94] [Memory Addr: c] [] [] [] [] [] [] [] []
Incident details in: oracle_base>/crs/<nodename>/crs/incident/incdir_9/ologgerd_i9.trc

2018-11-16 03:24:19.571 [CSSDMONITOR(15735)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 15735
2018-11-16 03:24:19.738 [EVMD(15737)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 15737
2018-11-16 03:24:20.882 [CSSDAGENT(15765)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 15765
2018-11-16 03:24:22.278 [OCSSD(15822)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 15822
2018-11-16 03:24:23.654 [OCSSD(15822)]CRS-1713: CSSD daemon is started in hub mode

Dump continued from file: <oracle_base>/crs/<nodename>/crs/trace/ologgerd.trc
[TOC00001]
CRS-8503 [crfoci_getconnstr()+532] [Signal/Exception: 11] [Instruction Addr: fffffffece392e94] [Memory Addr: c] [] [] [] [] [] [] [] []

 Error Descriptor: CRS-8503 [crfoci_getconnstr()+532] [Signal/Exception: 11] [Instruction Addr: fffffffece392e94] [Memory Addr: c] [] [] [] [] [] [] [] []
Error class: 0
Problem Key # of args: 1
Number of actions: 10
----- Incident Context Dump -----
Address: 0x1057fe908
Incident ID: 9
Problem Key: CRS 8503 [crfoci_getconnstr()+532]

Error: CRS-8503 [crfoci_getconnstr()+532] [Signal/Exception: 11] [Instruction Addr: fffffffece392e94] [Memory Addr: c] [] [] [] [] [] [] [] []
[00]: dbgexPopulateIncCtx [diag_dde]
[01]: dbgexProcessError [diag_dde]
[02]: dbgePostErrorDirectVaList_int [diag_dde]
[03]: dbgePostErrorDirect [diag_dde]
[04]: clsdAdr_process_CRS_8503 []
[05]: clsdAdrPostError []
[06]: clsdadrpr_CreateIncidentCheck []
[07]: clsdadrprAlert []
[08]: clsd_malertprintftSig []
[09]: clsbSigErrCB []
[10]: skgesig_sigactionHandler []
[11]: __sighndlr []
[12]: call_user_handler []
[13]: sigacthandler []
[14]: crfoci_getconnstr []<-- Signaling
[15]: crfoci_conn_create []
[16]: crflogdbora_init_conn_stmt []
[17]: crflogdbora_init_calog []
[18]: crflogd_calogwriter []
[19]: _lwp_start []

 

2018-11-16 03:21:15.739 : CRFOCI:12: crfoci_conn_create: Trying to connect to (DESCRIPTION = (CONNECT_TIMEOUT=60) (ADDRESS = (PROTOCOL = tcp)(HOST = 10.208.156.248)(PORT = 1525))(CONNECT_DATA = (SERVICE_NAME = gimr_dscrep_10))).
2018-11-16 03:22:10.052 : CRFLOGD:15: Database writer queue size for nview is high(200) Sending STOPSEND message to sysmond on <nodename> node.
2018-11-16 03:22:11.031 : CRFLOGD:17: lsys: Loggerd cannot accept any more packets. Sending STOPSEND message to sysmond on<nodename> node.
2018-11-16 03:22:11.033 : CRFLOGD:18: lsys: Loggerd cannot accept any more packets. Sending STOPSEND message to sysmond on <nodename> node.
2018-11-16 03:22:11.039 : CRFLOGD:16: lsys: Loggerd cannot accept any more packets. Sending STOPSEND message to sysmond on <nodename> node.
2018-11-16 03:22:15.755 : CRFOCI:12: crfoci_checkerr: Error - OCI_ERROR ORA-12170: TNS:Connect timeout occurred

2018-11-16 03:22:15.755 : CRFOCI:12: crfoci_conn_create: Trying to connect to (DESCRIPTION = (CONNECT_TIMEOUT=60) (ADDRESS = (PROTOCOL = tcp)(HOST = 10.208.160.244)(PORT = 1525))(CONNECT_DATA = (SERVICE_NAME = gimr_dscrep_10))).
2018-11-16 03:22:16.031 : CRFOCI:10: crfoci_conn_create: Trying to connect to (DESCRIPTION = (CONNECT_TIMEOUT=60) (ADDRESS = (PROTOCOL = tcp)(HOST = 10.208.156.248)(PORT = 1525))(CONNECT_DATA = (SERVICE_NAME = gimr_dscrep_10))).
2018-11-16 03:22:44.678 : CRFLOGD:19: UI update: crfm_receive rc=-2
2018-11-16 03:22:44.678 : CRFLOGD:19: Error in crflogui_update(), ret=-2
2018-11-16 03:23:17.043 : CRFOCI:10: crfoci_checkerr: Error - OCI_ERROR ORA-12170: TNS:Connect timeout occurred

2018-11-16 03:23:17.043 : CRFOCI:10: crfoci_conn_create: Trying to connect to (DESCRIPTION = (CONNECT_TIMEOUT=60) (ADDRESS = (PROTOCOL = tcp)(HOST = 10.208.160.244)(PORT = 1525))(CONNECT_DATA = (SERVICE_NAME = gimr_dscrep_10))).
2018-11-16 03:23:17.052 : CRFOCI:10: crfoci_checkerr: Error - OCI_ERROR ORA-12547: TNS:lost contact

2018-11-16 03:23:17.052 : CRFOCI:10: crfoci_conn_create: OCISessionGet failed
2018-11-16 03:23:17.052 : CRFLOGD:10: crflogdbora_init_conn_stmt: crfoci connection create failed 19
2018-11-16 03:23:17.052 : CRFLOGD:10: crflogdbora_init_conn_stmt: errbuf ORA-12547: TNS:lost contact

 ..

2018-11-16 03:23:18.334 : CRFM:8: crfm_register5: clssgsreg failed with 22
2018-11-16 03:23:18.334 : CRFLOGD:8: lsys: CSS registration failed for1 times
2018-11-16 03:23:23.332 : CRFM:8: crfm_register5: clssgsreg failed with 22
2018-11-16 03:23:23.332 : CRFLOGD:8: lsys: CSS registration failed for2 times
2018-11-16 03:23:27.949 : CRFOCI:14: crfoci_connmgr: Initilizing CSS
2018-11-16 03:23:27.949 : CRFOCI:14: crfoci_connmgr: Registering with MGMTDB group
2018-11-16 03:23:27.949 : CRFOCI:14: crfoci_mgmtregister: clssgsreg failed with retcode 22
2018-11-16 03:23:27.949 : CRFOCI:14: crfoci_connmgr: crfoci_mgmtregister failed with errcode 6
2018-11-16 03:23:28.332 : CRFM:8: crfm_register5: clssgsreg failed with 22
2018-11-16 03:23:28.332 : CRFLOGD:8: lsys: CSS registration failed for3 times
2018-11-16 03:23:28.958 : CRFLOGD:17: ***Thread lsys going down for <nodename>***
2018-11-16 03:23:33.332 : CRFM:8: crfm_register5: clssgsreg failed with 22
2018-11-16 03:23:33.332 : CRFLOGD:8: lsys: CSS registration failed for4 times
2018-11-16 03:23:33.333 :GIPCXCPT:8: gipcInternalSend: connection not valid for send operation endp 10592e950 [0000000000009768] { gipcEndpoint : localAddr 'ipc', remoteAddr 'ipc://<nodename>CRFM_SIPC', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1057fc8f0, sendp 101c92d10 status 13flags 0x2108061a, flags-2 0x0, usrFlags 0x20 }, ret gipcretConnectionInvalid (13)
2018-11-16 03:23:33.333 :GIPCXCPT:8: gipcSendSyncF [crfm_connect_timeout : crfm.c : 3376]: EXCEPTION[ ret gipcretConnectionInvalid (13) ] failed to send on endp 10592e950 [0000000000009768] { gipcEndpoint : localAddr 'ipc', remoteAddr 'ipc://<nodename>CRFM_SIPC', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1057fc8f0, sendp 101c92d10 status 13flags 0x2108061a, flags-2 0x0, usrFlags 0x20 }, addr 0000000000000000, buf 105cc01d0, len 366, flags 0x0
2018-11-16 03:23:33.333 : CRFM:8: crfm_connect_to: send fail(gipcret: 13)