AIA Connection Pool (JDBC) Leak, Inactive Sessions Growing and Dead Lock

(Doc ID 860742.1)

Last updated on OCTOBER 20, 2016

Applies to:

Oracle Application Integration Architecture Foundation Pack - Version 2.0.1 to 2.5 [Release 2]
Information in this document applies to any platform.

Symptoms

Database runs out of connections (maximum database connections in Oracle set to 500). SOA Suite 10.1.3.4 MLR5 with AIA 2.3 sole client to the database. The total OC4J managed JDBC connections pools reaches 500 with AIAConnectionPool using 400+ during moderate customer Development activities.

Steps To Reproduce:
Actively use a standard SOA/AIA 2.3 installation, AIAConnectionPool will eventually grow and not
release connections. Connections show inactive in the database.

Deadlocks were also detected on the JVM and also on the database.

Java-level deadlock:
=============================

"RMICallHandler-233":
waiting to lock monitor 0x49c10454 (object 0x98639f70, a oracle.jdbc.driver.T4CCallableStatement),
which is held by "RMICallHandler-79"
"RMICallHandler-79":
waiting to lock monitor 0x49c100d4 (object 0x98639d20, a oracle.jdbc.driver.T4CConnection),
which is held by "RMICallHandler-233"


Java stack information for the threads listed above:
===================================================

"RMICallHandler-233":
at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1340)
- waiting to lock <0x98639f70> (a oracle.jdbc.driver.T4CCallableStatement)
- locked <0x98639d20> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.PhysicalConnection.close_statements(PhysicalConnection.java:2333)
- locked <0x98639d20> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:1288)
- locked <0x98639d20> (a oracle.jdbc.driver.T4CConnection)
at oracle.apps.aia.core.eh.PublishMessageSession.publishFaultMessageOnTopic(Unknown Source)
at oracle.apps.aia.core.eh.AIAErrorHandlerImpl.sendNotification(Unknown Source)
at oracle.apps.aia.core.eh.BPELJavaAction.handleBPELFault(Unknown Source)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoveryActionJava.execute(RecoveryActionJava.java:81)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoverFault.recoverAndChain(RecoverFault.java:199)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoverFault.resolveAndRecover(RecoverFault.java:356)

"RMICallHandler-79":
at oracle.jdbc.driver.OraclePreparedStatement.setNullInternal(OraclePreparedStatement.java:3690)
- waiting to lock <0x98639d20> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleCallableStatement.setNull(OracleCallableStatement.java:4322)
- locked <0x98639f70> (a oracle.jdbc.driver.T4CCallableStatement)
at oracle.jms.AQjmsProducer.enqueue(AQjmsProducer.java:1069)
- locked <0x98641458> (a oracle.jms.AQjmsProducer)
at oracle.jms.AQjmsProducer.publish(AQjmsProducer.java:1407)
- locked <0x98641458> (a oracle.jms.AQjmsProducer)
at oracle.jms.AQjmsProducer.publish(AQjmsProducer.java:1254)
- locked <0x98641458> (a oracle.jms.AQjmsProducer)
at oracle.apps.aia.core.eh.PublishMessageSession.publishFaultMessageOnTopic(Unknown Source)
at oracle.apps.aia.core.eh.AIAErrorHandlerImpl.sendNotification(Unknown Source)
at oracle.apps.aia.core.eh.BPELJavaAction.handleBPELFault(Unknown Source)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoveryActionJava.execute(RecoveryActionJava.java:81)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoverFault.recoverAndChain(RecoverFault.java:199)
at com.collaxa.cube.engine.ext.wmp.faultpolicy.RecoverFault.resolveAndRecover(RecoverFault.java:356)

$SOA_HOME/opmn/logs/default_group~oc4j_soa~default_group~1.log showed:

09/06/04 05:07:14 oracle.jms.AQjmsException: ORA-00060: deadlock detected  
while waiting for resource  
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 6008  
ORA-06512: at line 1  
ORA-06512: at "SYS.DBMS_AQJMS", line 174  
ORA-06512: at line 1  
at oracle.jms.AQjmsSession.alterDurableSubscriber(AQjmsSession.java:3880)  
at oracle.jms.AQjmsSession.createSubInOldMode(AQjmsSession.java:2771)  
at oracle.jms.AQjmsSession.createDurableSubscriber(AQjmsSession.java:2400)  
at oracle.jms.AQjmsSession.createDurableSubscriber(AQjmsSession.java:2330)  
at oracle.jms.AQjmsSession.createDurableSubscriber(AQjmsSession.java:2175)  
at oracle.apps.aia.core.eh.PublishMessageSession.publishFaultMessageOnTopic(PublishMessageSession.java:175)  
......

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms