NMS Clients are Not Getting Updates or "ORA-00060: deadlock detected" on AMR_RESPONSES Table (Doc ID 2521599.1)

Last updated on APRIL 12, 2024

Applies to:

Symptoms

On : 2.3.0.2.0 version, Java Application Server

NMS system - can not access events

We did a redeploy of the cesejb.ear and the system no longer will display events. We also now get excessive time outs.
We suspect this is either caused by a database issue or an issue with our database cluster.

These errors appeared in the Web Call Entry client log:

2019-03-19 16:31:32,050 [AWT-EventQueue-1] WARN com.splwg.oms.client.util.proxy.ProxyInvocationHandler: invoke: InterSysOperations.reportIncidentSP elapsed: 180054 ms. req: 858 bytes, resp: 0 bytes
2019-03-19 16:31:32,050 [AWT-EventQueue-1] ERROR com.splwg.oms.client.callentry.SaveCommand: execute command SaveCommand
org.omg.CORBA.TIMEOUT: Response timed out after: '90000' milliseconds.
at weblogic.iiop.Utils.mapRemoteToCORBAException(Utils.java:715) ~[?:?]
at weblogic.iiop.Utils.mapToCORBAException(Utils.java:740) ~[?:?]
at weblogic.corba.idl.RemoteDelegateImpl.invoke(RemoteDelegateImpl.java:392) ~[?:?]
...
2019-03-19 16:37:43,276 [AWT-EventQueue-1] ERROR com.splwg.oms.client.BaseEnvironmentManager: We have lost our connection
org.omg.CORBA.COMM_FAILURE: Could not resolve object name: InterSys_nms_2
at com.splwg.oms.ejb.session.CorbaObjectContainer.getObject(CorbaObjectContainer.java:261) ~[?:?]

There might also be a number of deadlock errors in the *DBService logs:

-- ORACLE ERROR -- OCI return code from OCIStmtExecute is OCI_ERROR
while processing command 'DELETE FROM amr_responses WHERE request_idx = :request_idx'
ORA-00060: deadlock detected while waiting for resource
parse offset = 0
03/27/19 02:13:34: dbrdwr:165413 oms_oci::try_sql - deadlock detected, sleeping 3 seconds...
03/27/19 02:13:37: dbrdwr:165413 oms_oci::try_sql - Deadlock: going to execute

The managed server log could also have these errors:

2019-03-04 12:22:57,978 ERROR amr.AmrImpl      :712  [] - completeRequests: error updating AMR_RESPONSES - ORA-00060: deadlock detected while waiting for resource
2019-03-04 12:22:57,979 ERROR amr.AmrImpl      :715  [] - ORA-00060: deadlock detected while waiting for resource
2019-03-05 09:39:20,160 ERROR amr.AmrImpl      :712  [] - completeRequests: error updating AMR_RESPONSES - ORA-00060: deadlock detected while waiting for resource
2019-03-05 09:39:20,160 ERROR amr.AmrImpl      :715  [] - ORA-00060: deadlock detected while waiting for resource
2019-03-25 20:06:00,245 ERROR amr.AmrImpl      :712  [] - completeRequests: error updating AMR_RESPONSES - ORA-00060: deadlock detected while waiting for resource

This warning appeared in the Web Workspace client log:

2019-03-19 14:47:08,137 [Timer-1] WARN com.splwg.oms.client.util.proxy.ProxyInvocationHandler: invoke: ISession.getTroubleEvents elapsed: 77307 ms. req: 203 bytes, resp: 0 bytes

The issue can be reproduced at will with the following steps:
1. Launch Web Call Entry
2. Attempt to save a call.
3. The app hangs

or

1. Launch Web Workspace
2. Grab authority over a set of zones
3. WorkAgenda does not show any events


The AWR report showed an extremely long-running update that seemed stuck:

The SQL text is a simple DELETE command:

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.