My Oracle Support Banner

11g Grid Control Performance: OMS Crashes if Repository Database is Unresponsive / Hung (Doc ID 1287904.1)

Last updated on NOVEMBER 15, 2019

Applies to:

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Information in this document applies to any platform.

Symptoms

-  The <EM_INSTANCE_HOME>/sysman/log/emoms.trc shows:

2011-03-23 08:09:56,193 [HealthMonitor] WARN healthMonitor.HealthMonitor run.371 - HealthMonitor : Found errant task : TaskRegn:ID1,Callback:class oracle.sysman.emdrep.jobs.Dispatcher,Iterative:true,Duration:300,DueTime:1300012076191:Callback Details:oracle.sysman.emdrep.jobs.Dispatcher@3697a8c1
2011-03-23 08:09:56,195 [HealthMonitor] ERROR healthMonitor.HealthMonitor restart.531 - HealthMonitor Mar 23, 2011 08:09:56 AM Job Dispatcher error: Job Dispatcher timed out. Job Dispatcher id=4
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
/oracleoem1110/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

The HealthMonitor thread has attempted to restart the OMS due to Job Dispatcher timeout. This can occur for any other critical Java thread of the OMS as well.

-  Attempting to start the OMS after this fails. The <EM_INSTANCE_HOME>/sysman/log/emctl.log shows:

java.lang.Exception: Encountered error while hitting page. Status code is 503
at oracle.sysman.emctl.oms.StatusOMSCmd.processStatusOMS(StatusOMSCmd.java:223)
at oracle.sysman.emctl.wls.OMSController.statusOMS(OMSController.java:897)
at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:511)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:219)
2011-01-19 09:45:01,715 [Main Thread] INFO wls.OMSController statusOMS.980 - statusOMS finished with result: 9
2011-01-19 09:45:01,715 [Main Thread] INFO wls.OMSController startOMS.636 - Issuing net start for oms service...
2011-01-19 09:45:01,715 [Main Thread] INFO wls.OMSController startOMS.643 - net start cmd is [C:\WINDOWS\system32\net.exe, start, OracleManagementServer_EMGC_OMS1_1]
2011-01-19 09:45:01,762 [Thread-2] INFO wls.OMSController run.1159 - <ERR>The requested service has already been started.
2011-01-19 09:45:01,762 [Thread-2] INFO wls.OMSController run.1159 - <ERR>
2011-01-19 09:45:01,762 [Thread-2] INFO wls.OMSController run.1159 - <ERR>More help is available by typing NET HELPMSG 2182.
2011-01-19 09:45:01,762 [Thread-2] INFO wls.OMSController run.1159 - <ERR>
2011-01-19 09:45:01,762 [Main Thread] INFO wls.OMSController processCmd.1018 - wlst process is finished
2011-01-19 09:45:01,762 [Main Thread] INFO wls.OMSController processCmd.1020 - wlst process exited with code 2
2011-01-19 09:45:01,777 [Main Thread] INFO wls.OMSController processCmd.1022 - wlst outstream joined
2011-01-19 09:45:01,777 [Main Thread] INFO wls.OMSController processCmd.1024 - wlst errstream joined
2011-01-19 09:45:01,777 [Main Thread] INFO wls.OMSController startOMS.645 - net start return code is 2
2011-01-19 09:45:01,777 [Main Thread] ERROR wls.OMSController startOMS.648 - Windows service OracleManagementServer_EMGC_OMS1_1 failed to be started
2011-01-19 09:45:01,777 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.140 - HTTP port in emgc property file: 7201
2011-01-19 09:45:01,777 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.141 - HTTPS port in emgc property file: 7301
2011-01-19 09:45:01,777 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.157 - Using http port.
2011-01-19 09:45:01,777 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.197 - Attempting to connect to http://<omsmachine.domain>:7201/em/console/logon/logon
2011-01-19 09:45:01,793 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.202 - page status code is 503
2011-01-19 09:45:01,793 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.204 - header is 3
2011-01-19 09:45:01,793 [Main Thread] DEBUG oms.StatusOMSCmd processStatusOMS.222 - Encountered error while hitting page. Status code is 503
2011-01-19 09:45:01,809 [Main Thread] ERROR oms.StatusOMSCmd processStatusOMS.234 - Encountered error while hitting page. Status code is 503
java.lang.Exception: Encountered error while hitting page. Status code is 503
at oracle.sysman.emctl.oms.StatusOMSCmd.processStatusOMS
(StatusOMSCmd.java:223)
at oracle.sysman.emctl.wls.OMSController.statusOMS(OMSController.java:897)
at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:657)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:219)
2011-01-19 09:45:01,809 [Main Thread] INFO wls.OMSController statusOMS.980 - statusOMS finished with result: 9

-  The 'emctl status oms' command returns:
E:\Oracle\Middleware\oms11g\BIN>emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

-  The <Domain Home>/EMGC_OMS1/logs/EMGC_OMS1.out shows:

<Mar 23, 2011 8:10:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:13:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:16:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:19:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:22:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:25:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>
<Mar 23, 2011 8:28:27 AM EDT> <Error> <Health> <BEA-310001> <Subsystem JDBC has failed. Setting server state to Failed.>

- Attempting to connect to the Repository Database as 'sysman' user via sqlplus hangs.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.