CRS-8503: Oracle Clusterware OSYSMOND process with operating system process ID <pid> experienced fatal signal or exception code 11
(Doc ID 1928851.1)
Last updated on OCTOBER 04, 2024
Applies to:
Oracle Database Cloud Schema Service - Version N/A and laterOracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
12.1.0.2 Grid Infrastructure process OSYSMOND crashes every hour and a core file is generated in <ORACLE_BASE>/diag/crs/<node>/crs/cdump/core_<pid>
<ORACLE_BASE>/diag/crs/<node>/crs/trace/alert.log
2014-09-15 02:42:23.888 [OSYSMOND(22360)]CRS-8503: Oracle Clusterware OSYSMOND process with operating system process ID 22360 experienced fatal signal or exception code 11
Sweep [inc][3497]: completed
2014-09-15 02:42:24.084 [OSYSMOND(5135)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 5135
Mon Sep 15 03:42:25 2014
Errors in file /home/grid/app/grid/diag/crs/racnode1/crs/trace/osysmond.trc (incident=3505):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /home/grid/app/grid/diag/crs/racnode1/crs/incident/incdir_3505/osysmond_i3505.trc
....
2014-09-15 03:42:25.240 [OSYSMOND(5135)]CRS-8503: Oracle Clusterware OSYSMOND process with operating system process ID 5135 experienced fatal signal or exception code 11
Sweep [inc][3505]: completed
2014-09-15 03:42:25.473 [OSYSMOND(2725)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 2725
Mon Sep 15 04:42:26 2014
Errors in file /home/grid/app/grid/diag/crs/racnode1/crs/trace/osysmond.trc (incident=3513):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /home/grid/app/grid/diag/crs/racnode1/crs/incident/incdir_3513/osysmond_i3513.trc
Sweep [inc][3497]: completed
2014-09-15 02:42:24.084 [OSYSMOND(5135)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 5135
Mon Sep 15 03:42:25 2014
Errors in file /home/grid/app/grid/diag/crs/racnode1/crs/trace/osysmond.trc (incident=3505):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /home/grid/app/grid/diag/crs/racnode1/crs/incident/incdir_3505/osysmond_i3505.trc
....
2014-09-15 03:42:25.240 [OSYSMOND(5135)]CRS-8503: Oracle Clusterware OSYSMOND process with operating system process ID 5135 experienced fatal signal or exception code 11
Sweep [inc][3505]: completed
2014-09-15 03:42:25.473 [OSYSMOND(2725)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 2725
Mon Sep 15 04:42:26 2014
Errors in file /home/grid/app/grid/diag/crs/racnode1/crs/trace/osysmond.trc (incident=3513):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /home/grid/app/grid/diag/crs/racnode1/crs/incident/incdir_3513/osysmond_i3513.trc
<ORACLE_BASE>/diag/crs/<node>/crs/trace/osysmond.trc
2014-09-15 02:42:23.863422 : CRFMOND:1977382656: asmdskname <DISK>, asmpath ORCL:OCRVOT_<PATH>
Trace file /home/grid/app/grid/diag/crs/<node>/crs/trace/osysmond.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
Incident 3425 created, dump file: /home/grid/app/grid/diag/crs/<node>/crs/incident/incdir_3425/osysmond_i3425.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
CLSB:1977382656: Oracle Clusterware infrastructure error in OSYSMOND (OS PID 39387): Fatal signal 11 has occurred in program osysmond thread 1977382656; nested signal count is 1
Trace file /home/grid/app/grid/diag/crs/<node>/crs/trace/osysmond.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
CLSB:520263264: Argument count (argc) for this daemon is 1
CLSB:520263264: Argument 0 is: /dba/app/grid/12.1.0.2/bin/osysmond.bin
2014-09-15 02:42:24.076820 : default:520263264: clsmond_main: Successfully initilized enviornment variable
2014-09-15 02:42:24.076824 : default:520263264: clsmond_main: Successfully initilized lpm
Trace file /home/grid/app/grid/diag/crs/<node>/crs/trace/osysmond.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
Incident 3425 created, dump file: /home/grid/app/grid/diag/crs/<node>/crs/incident/incdir_3425/osysmond_i3425.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
CLSB:1977382656: Oracle Clusterware infrastructure error in OSYSMOND (OS PID 39387): Fatal signal 11 has occurred in program osysmond thread 1977382656; nested signal count is 1
Trace file /home/grid/app/grid/diag/crs/<node>/crs/trace/osysmond.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
CLSB:520263264: Argument count (argc) for this daemon is 1
CLSB:520263264: Argument 0 is: /dba/app/grid/12.1.0.2/bin/osysmond.bin
2014-09-15 02:42:24.076820 : default:520263264: clsmond_main: Successfully initilized enviornment variable
2014-09-15 02:42:24.076824 : default:520263264: clsmond_main: Successfully initilized lpm
<ORACLE_BASE>/diag/crs/<node>/crs/incident/incdir_<id>/osysmond_i<id>.trc
Problem Key: CRS 8503
Error: CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
[00]: dbgexProcessError [diag_dde]
[01]: dbgeExecuteForError [diag_dde]
[02]: dbgePostErrorDirect [diag_dde]
[03]: clsdAdrPostError []<-- Signaling
[04]: clsbSigErrCB []
[05]: skgesig_sigactionHandler []
[06]: __sighandler []
[07]: kfodpdGetKey []
[08]: crfmond_discover []
[09]: crfmond_mark_asmdisks []
[10]: crfmond_refresh []
[11]: start_thread []
Error: CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
[00]: dbgexProcessError [diag_dde]
[01]: dbgeExecuteForError [diag_dde]
[02]: dbgePostErrorDirect [diag_dde]
[03]: clsdAdrPostError []<-- Signaling
[04]: clsbSigErrCB []
[05]: skgesig_sigactionHandler []
[06]: __sighandler []
[07]: kfodpdGetKey []
[08]: crfmond_discover []
[09]: crfmond_mark_asmdisks []
[10]: crfmond_refresh []
[11]: start_thread []
OS messages file on Linux
Sep 5 05:07:26 lvicoraracq01 Oracle Clusterware: 2014-09-05 05:07:25.886 [(3715)]CRS-8503: Oracle Clusterware OSYSMOND process with operating system process ID 3715 experienced fatal signal or exception code 11
Sep 5 05:07:26 lvicoraracq01 abrt[15784]: Saved core dump of pid 3715 (/u01/app/crs/product/12.1.0/grid/bin/osysmond.bin) to /var/spool/abrt/ccpp-2014-09-05-05:07:25-3715 (37625856 bytes)
Sep 5 05:07:26 lvicoraracq01 abrtd: Directory 'ccpp-2014-09-05-05:07:25-3715' creation detected
Sep 5 05:07:26 lvicoraracq01 abrtd: Executable '/u01/app/crs/product/12.1.0/grid/bin/osysmond.bin' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 5 05:07:26 lvicoraracq01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-09-05-05:07:25-3715' exited with 1
Sep 5 05:07:26 lvicoraracq01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-09-05-05:07:25-3715'
Sep 5 05:07:26 lvicoraracq01 Oracle Clusterware: 2014-09-05 05:07:26.248#012[(15791)]CRS-8500:Oracle Clusterware OSYSMOND process is starting with operating system process ID 15791
Sep 5 05:07:26 lvicoraracq01 abrt[15784]: Saved core dump of pid 3715 (/u01/app/crs/product/12.1.0/grid/bin/osysmond.bin) to /var/spool/abrt/ccpp-2014-09-05-05:07:25-3715 (37625856 bytes)
Sep 5 05:07:26 lvicoraracq01 abrtd: Directory 'ccpp-2014-09-05-05:07:25-3715' creation detected
Sep 5 05:07:26 lvicoraracq01 abrtd: Executable '/u01/app/crs/product/12.1.0/grid/bin/osysmond.bin' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 5 05:07:26 lvicoraracq01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-09-05-05:07:25-3715' exited with 1
Sep 5 05:07:26 lvicoraracq01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-09-05-05:07:25-3715'
Sep 5 05:07:26 lvicoraracq01 Oracle Clusterware: 2014-09-05 05:07:26.248#012[(15791)]CRS-8500:Oracle Clusterware OSYSMOND process is starting with operating system process ID 15791
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |