12c Cloud Control Performance: OMS Re-starts Regularly Due to "Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat"

(Doc ID 1424303.1)

Last updated on OCTOBER 22, 2013

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.
Checked for relevance on 22-Oct-2013

Symptoms

A newly installed 12.1 OMS crashes and re-starts by itself regularly, sometimes once every day.

- The <EM_INSTANCE_BASE>/em/EMGC_OMS1/sysman/log/emctl.msg:

HealthMonitor Feb 2, 2012 2:50:27 PM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/Middleware_OEM/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

-  The <EM_INSTANCE_BASE>/em/EMGC_OMS1/sysman/log/emoms_pbs.trc:

2012-02-06 08:05:23,869 [PbsOMSHeartbeatThread] ERROR healthMonitor.HealthMonitor executeCommand.933 - HealthMonitor : Executing diagnostic command for module omsThread. Feb 6, 2012 8:05:23 AM

2012-02-06 08:05:55,935 [PbsOMSHeartbeatThread] ERROR healthMonitor.HealthMonitor restart.662 - HealthMonitor Feb 6, 2012 8:05:22 AM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/Middleware_OEM/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

- The <EM_INSTANCE_BASE>/em/EMGC_OMS1/sysman/log/oms_diag_info_2012_02_06_08_05_48.msg shows:

PbsOMSHeartbeatThread-85 | 3245 | getConnection | 2012-02-06 07:58:28 | 2012-02-06 07:58:28 | 0 ms
PbsOMSHeartbeatThread-85 | 3246 | mainTask | 2012-02-06 08:05:22 | - | Method running for 26419 ms
PbsOMSHeartbeatThread-85 | 3246 | heartbeat | 2012-02-06 08:05:22 | - | Method running for 26388 ms
PbsOMSHeartbeatThread-85 | 3246 | getConnection | 2012-02-06 08:05:22 | 2012-02-06 08:05:22 | 0 ms


For almost 7 minutes, the PbsOMSHeartbeatThread did not get any CPU cycles for updating the OMS heartbeat into the repository table.

- The <EM_INSTANCE_BASE>/em/EMGC_OMS1/sysman/log/repo_dump_2012_02_06_08_05_48.html shows that there are no blocking locks on the MGMT_FAILOVER_TABLE affecting new sessions from updating the table, which indicates that there was no problem at the repository database level.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms