My Oracle Support Banner

Problem - Grid Control Agent Restarts with Core Dump when it Reaches its Maximum Thread Model Limit (Doc ID 357448.1)

Last updated on NOVEMBER 09, 2019

Applies to:

Enterprise Manager Base Platform - Version 10.1.0.2 to 10.2.0.5 [Release 10.1 to 10.2]
Information in this document applies to any platform.
Checked for relevance on 31-Jan-2014

Symptoms

Running Grid 10.1.0.4 Agent,  receiving email notification stating that the agent went down and then a few seconds later stating that the agent has come back up.  This can occur very frequently,  every 20 minutes,  or every few hours, depending on the load on the node.  If you check the status of the agent, it appears to be running fine,  uploads are occurring.  The ORACLE_HOME/sysman/log/  files show the agent core dumping, but no core file is produced:

ERRORS with agent in WARN (default) level of debugging:

emagent.nohup:

---- <TIMESTAMP>::Abnormality reported for EMAgent : 4276 -----
----- <TIMESTAMP>::Debugging component EMAgent -----
----- <TIMESTAMP>::generate first core file <AGENT_HOME>/sysman/emd/emagent.dmp for diagnosis -----
----- <TIMESTAMP>::generate second core file <AGENT_HOME>/sysman/emd/emagent.dmp for diagnosis -----
----- <TIMESTAMP>::generate second core file <AGENT_HOME>/sysman/emd/emagent.dmp for diagnosis -----
----- Attempting to kill EMAgent : 4276 -----
Attempt to free unreferenced scalar at <AGENT_HOME>\bin\emwd.pl line 702.
Signaling abnormal exit -----
----- <TIMESTAMP>::EMAgent exited at <TIMESTAMP> with signal 99 -----
----- <TIMESTAMP>::EMAgent has exited due to an internal error -----
----- <TIMESTAMP>:: - checking for corefile at $ORACLE_HOME\gc_agent\usfldbb1/sysman/emd -----


emagent.trc:

<TIMESTAMP> Thread-5256 ERROR HealthMonitor: HealthMonitor timeout because entry: ping
<TIMESTAMP> Thread-5256 ERROR HealthMonitor: Duetime passed current time in entry: ping
....
<TIMESTAMP> Thread-2256 ERROR scheduler: SchedEntry{oracle_database:<sid>:wait_sess_cls_10i+system_response_time_per_call_10i+service_10i} exceed next scheduletime, delay=901
<TIMESTAMP> Thread-2256 ERROR scheduler: SchedEntry{oracle_listener:<listener name>:Load+General Status} exceed next scheduletime, delay=901
<TIMESTAMP> Thread-2256 ERROR scheduler: SchedEntry{oracle_database:<sid_name>:alert_log_rollup} exceed next scheduletime, delay=900
<TIMESTAMP> Thread-2256 ERROR scheduler: failed to create thread for 1202 seconds, will exit


With the Agent set to trace in DEBUG mode for tracelevel.ThreadPool we see the following:

Thread-932 DEBUG ThreadPool: Max thread count 40 reached
2006-02-08 13:27:21 Thread-932 INFO  scheduler: Scheduler will wait for 3 seconds ...
2006-02-08 13:27:24 Thread-3140 DEBUG ThreadPool: runTask create a thread: System=10, Worker=40, Total=50
2006-02-08 13:27:24 Thread-3492 DEBUG ThreadPool: nmttp_run enter
2006-02-08 13:27:55 Thread-1564 INFO  main: ******************************************************************************
2006-02-08 13:27:55 Thread-1564 INFO  main: Starting Agent 10.1.0.4.1 from D:\oracle\product\10.1.0\gc_agent
2006-02-08 13:27:55 Thread-1564 INFO  main: ******************************************************************************
2006-02-08 13:27:55 Thread-1564 INFO  main: Initialize JVM capability and signal handlers
2006-02-08 13:27:55 Thread-1564 INFO  main: start thread pool manager
2006-02-08 13:27:55 Thread-1564 INFO  ThreadPool: User has neither specified a thread cap nor thread model to use
2006-02-08 13:27:55 Thread-1564 INFO  ThreadPool: TPM defaulting to SMALL model with 40 threads
2006-02-08 13:27:55 Thread-1564 DEBUG ThreadPool: default stack size of set 0
2006-02-08 13:27:55 Thread-1564 INFO  main: start health monitor

Changes

If the Agent was working fine previously and all of a sudden this has started to occur, then it may be getting caused by an increased load on the machine, for example:

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.