EM 12c : Enterprise Manager 12c Cloud Control OMS Crashed Suddenly with Error : Could not connect to repository (Doc ID 1615511.1)

Last updated on MARCH 19, 2015

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

12.1.0.3 OMS went down.

Error observed in the .../gc_inst/sysman/log/emctl.msg file:

HealthMonitor Dec 23, 2013 4:46:03 AM Job Dispatcher error: Could not connect to repository
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

 

Error observed in the .../gc_inst/sysman/log/emoms.trc file:

2013-12-23 04:39:10,008 [#EM_SYSTEM_POOL#:BadSQLEvaluation] WARN  jdbc.ConnectionCache _getConnection.380 - IO Error: The Network Adapter could not establish the connection
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:489)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:234)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:404)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:288)

 

During the timestamp when the error of crash observed in the OMS files, the following ORA-609 error is observed in the DB alert log:

Mon Dec 23 04:34:29 2013

Fatal NI connect error 12170.

 VERSION INFORMATION:
TNS for Linux: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
 Time: 23-DEC-2013 04:34:29
 Tracing not turned on.
 Tns error struct:
   ns main err code: 12535
   
TNS-12535: TNS:operation timed out
   ns secondary err code: 12606
   nt main err code: 0
   nt secondary err code: 0
   nt OS err code: 0
Errors in file /n01/oraadmin1/diag/rdbms/omrp03a/omrp03a1/trace/omrp03a1_smon_3534.trc:
ORA-21780: Maximum number of object durations exceeded.
 Client address: <unknown>
opiodr aborting process unknown ospid (15558) as a result of ORA-609

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms