Critical Error Err=3 Detected In Module Oms Heartbeat Recorder:Oms Will Be Restarted. (Doc ID 1359693.1)

Last updated on MARCH 26, 2016

Applies to:

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Information in this document applies to any platform.
Checked for relevance on 26th March' 16

Symptoms

11.1.0.1 Grid Control Crashes intermittently.

Errors observed from emctl.msg:

HealthMonitor Sep 16, 2011 12:48:18 PM OMS Heartbeat Recorder:  error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder: 
OMS will be restarted. A full thread dump will be generated in the log file 
/app/oracle/Middleware/oms11g/user_projects/domains/GCDomain/
servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem. 
Please consult My Oracle Support Note 964469.1 for additional information.
Errors observed from EMGC_OMS*.out* file:

<Sep 27, 2011 5:52:04 AM EDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '153' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "647" seconds working on the request
"weblogic.servlet.internal.ServletRequestImpl@33bc7536[
 POST
 /em/upload?type=metadata&EMD_URL=https%3a%2f%2ftx2spora04%3a3872%2femd%2fmain&
 X-ORCL-EMOV=4%2e0%2e0&X-ORCL-EMCV=10%2e2%2e0%2e4%2e0&X-ORCL-EMSV=10%2e2%2e0%2e
 4%2e0 HTTP/1.1
 X-ORCL-EMUR: https://hostname:3872/emd/main
 X-ORCL-EMAK: F5672E35F5D39E0F96A920FFB13A88F3
 Content-Length: 665716
 ECID-Context: 1.0000itPG^NSBLAq_4LnZ6i00068T000JHK;kXjE1ZDLIPHGj5PRj5SSgVPO^G
 Connection: Keep-Alive
 Proxy-Client-IP: 171.186.221.132
 X-Forwarded-For: 171.186.221.132
 X-WebLogic-KeepAliveSecs: 30
 X-WebLogic-Force-JVMID: 1636863620
 .
 ]", which is more than the configured time (StuckThreadMaxTime) of "600"
 seconds. Stack trace:
 .....
 "OMSHeartbeatThread" daemon prio=3 tid=0x00000001055e6800 nid=0x5f waiting on
 condition [0xffffffff07afe000]
 java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)


Also as per the ThreadDump generated in EMGC_OMS1.out file

"PingHeartBeatRecorder" daemon prio=3 tid=0x0000000103105800 nid=0x65
 runnable [0xffffffff040fe000]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at oracle.net.ns.Packet.receive(Packet.java:293)
 at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
 at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
 at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
 at oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
 at oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
 at oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
 at
 oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputSt
 reamWrapper.java:122)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms