12c Cloud Control: OMS is Re-started Frequently by the HealthMonitor Due to Problems in the Repository Database (Doc ID 1491412.1)

Last updated on AUGUST 19, 2015

Applies to:

Symptoms

The 12c OMS is crashing frequently and the following error messages are recorded in log files:

<ORACLE_BASE>/gc_inst/em/EMGC_OMS1/sysman/log/emctl.msg shows:

HealthMonitor Aug 27, 2012 8:23:07 PM OMS Heartbeat Recorder error: Heartbeat recorder timed out. OMS id=77
Critical error err=3 detected in module OMS Heartbeat Recorder
OMS will be restarted. A full thread dump will be generated in the log file
/pg1as/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

HealthMonitor Aug 27, 2012 11:37:13 PM OMS Heartbeat Recorder:  error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

HealthMonitor Aug 29, 2012 7:22:39 PM Job Dispatcher error: Job Dispatcher timed out. Job Dispatcher id=80
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
/pg1as/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

At each OMS re-start, the Repository Database's alert.log shows the below entries:

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
 Time: 30-AUG-2012 09:20:59
 Tracing not turned on.
 Tns error struct:
   ns main err code: 12537
   
TNS-12537: TNS:connection closed
   ns secondary err code: 12560
   nt main err code: 0
   nt secondary err code: 0
   nt OS err code: 0
opiodr aborting process unknown ospid (28979) as a result of ORA-609


Wed Aug 29 01:14:46 2012
Errors in file /u01/app/oracle/diag/rdbms/omr/OMR/trace/OMR_cjq0_16843.trc  (incident=45073):
ORA-00445: background process "J011" did not start after 120 seconds
Incident details in: /u01/app/oracle/diag/rdbms/omr/OMR/incident/incdir_45073/OMR_cjq0_16843_i45073.trc
Wed Aug 29 01:14:46 2012
Dumping diagnostic data in directory=[cdmp_20120829011446], requested by (instance=1, osid=16843 (CJQ0)), summary=[incident=45073].
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/app/oracle/diag/rdbms/omr/OMR/trace/OMR_cjq0_16843.trc:

Cause