EM 11g: Enterprise Manager 11g Grid Control OMS Fails to Start, SYSMAN SESSIONS COUNT REACHES MAXIMUM / SYSMAN ACCOUNT LOCKED

(Doc ID 1339990.1)

Last updated on JANUARY 05, 2018

Applies to:

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Information in this document applies to any platform.

Symptoms

Scenario 1:

Enterprise Manager (EM) 11.1 Grid Control OMS fails to start with the following error:

./emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Could Not Be Started
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.
Please check
EM_Instance_HOME/em/EMGC_OMS1/sysman/log/emctl.log for error details

Sessions are hanging in repository db on 'library cache lock'.
Session limit is quickly reached
Listener status shows fine.
rcuJDBC engine is able to establish connection to repository database (see <note:1228904.1> "11g Grid Control: How to Verify the Connectivity from 11g OMS to Repository Database using rcuJDBCEngine"

When trying to start the OMS , the number of sessions on the database keeps growing and when it reaches the max limit for sessions, the repository database becomes hung.

Example of the increase in sessions:-

SQL> select username, count(*) from v$session group by username;

USERNAME COUNT(*)
------------------------------ ----------
624
SYS 11
SYSMAN_MDS 1

SQL> /

USERNAME COUNT(*)
------------------------------ ----------
626
SYS 11
SYSMAN_MDS 1

SQL> /

USERNAME COUNT(*)
------------------------------ ----------
766
SYS 11
SYSMAN_MDS 1

Errors observed from ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out:

ConnectionService.init: Get repos properties from credential store.
Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = ORA-01017: invalid username/password; logon denied

Invalid Connection Pool. ERROR = ORA-01017: invalid username/password; logon denied

From the above errors its clear that the SYSMAN credentials were not getting retrived from Credential Store.

Errors observed from repository database alert.log ($ORACLE_BASE/diag/rdbms/<sid>/<sid>/trace):

inbound connections are timing out.

TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=<IPAddress>)(PORT=61034))
WARNING: inbound connection timed out (ORA-3136)

ADDM report gives the following:

Activity During the Analysis Period

Total database time was 3756385 seconds.
The average number of active sessions was 1646.82.

Summary of Findings

Description   Active Sessions|Recommendations Percent of Activity

1 Session Connect and Disconnect   1646.8 | 100      1
2 Shared Pool Latches             1645.11 | 99.9     0

Scenario II:-

OMS does not start and accessing the console gives the following message:

"OMS is unavailable. Backend WLS or EM application seems to be down"

Errors observed from ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out:-

Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = Io exception: Connection reset
<Jul 6, 2011 11:23:45 AM EDT> <Warning> <JDBC> <BEA-001110> <No test table set up for pool "mds-sysman_mds". Connections will not be tested.>
Invalid Connection Pool. ERROR = Io exception: Connection reset

Invalid Connection Pool. ERROR = Io exception: Connection reset
Invalid Connection Pool. ERROR = Got minus one from a read call

<Jul 6, 2011 11:48:58 AM EDT> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "mds-sysman_mds": Got minus one from a read call>

Since the credential store details are not stored properly, it can be observed that the number of sessions keep increasing.

Errors observed from..gc_inst/em/EMGC_OMS1/sysman/log/emctl.log:

2011-07-15 12:48:42,192 [main] DEBUG oms.StatusOMSCmd processStatusOMS.204 - header is 3
2011-07-15 12:48:42,193 [main] DEBUG oms.StatusOMSCmd processStatusOMS.222 - Encountered error while hitting page. Status code is 503
2011-07-15 12:48:42,193 [main] ERROR oms.StatusOMSCmd processStatusOMS.234 - Encountered error while hitting page. Status code is 503
java.lang.Exception: Encountered error while hitting page. Status code is 503
at oracle.sysman.emctl.oms.StatusOMSCmd.processStatusOMS(StatusOMSCmd.java:223)
at oracle.sysman.emctl.wls.OMSController.statusOMS(OMSController.java:897)
at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:617)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:219)
2011-07-15 12:48:42,200 [main] INFO wls.OMSController statusOMS.980 - statusOMS finished with result: 9

Errors observed from repository database alert.log ($ORACLE_BASE/diag/rdbms/<sid>/<sid>/trace):-

Fatal NI connect error 12170.
.
.
Tns error struct:
ns main err code: 12535

TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=<IPAddress>)(PORT=46999))
WARNING: inbound connection timed out (ORA-3136)

Scenario III:

OMS fails to start with following error:

\Oracle\Middleware\oms11g\BIN>emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

Connecting to the repository database via sql*plus using SYSMAN is hanging / throwing error:

oracle@s-lim01-ogc01 ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.2.0 Production on Tue Jul 19 17:44:05 2011
Copyright (c) 1982, 2010, Oracle. All rights reserved.

SQL> connect sysman
Enter password:

(hangs)

SQL> connect sysman
Enter password:
ERROR:
ORA-00020: maximum number of processes (1000) exceeded

Errors observed from ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logsEMGC_OMS1.out:

Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Invalid Connection Pool. ERROR = ORA-28000: the account is locked

SQL> select USERNAME,ACCOUNT_STATUS from dba_users where username='SYSMAN';

USERNAME ACCOUNT_STATUS
---------------------------

SYSMAN LOCKED

Changes

The SYSMAN password was changed on the repository database

Cause

	Sign In with your My Oracle Support account
	Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms