JDBC Leak Detection in Oracle Glassfish Server Can Lock the Connection Pool When Attempting to Reclaim an Active Connection (Doc ID 1281608.1)

Last updated on NOVEMBER 05, 2016

Applies to:

Oracle GlassFish Server - Version 2.1.1 and later
Information in this document applies to any platform.
***Checked for relevance on 15-Aug-2013***

Symptoms

This problem results in threads blocking on a JDBC Connection Pool when the connection pool is trying to reclaim a JDBC connection that has exceeded the pool's leak detection timeout setting. 

While the connection pool is locked, no other request threads can either obtain or return connections to the pool.  This results in those requests appearing to hang while they wait to access the pool.

This problem is best seen in a Java thread dump taken at the point the hang has been noticed.  Please refer to <Note:1213467.1> for details on how to capture information for a hung application server that includes information on capturing a Java thread dump.

The first symptom is a report of a potential leaked connection, similar to this:

[#|2011-01-14T17:36:30.892+0000|WARNING|sun-appserver2.1.1|javax.enterprise.resource.resourceadapter|_ThreadID=15;_ThreadName=Timer-5;ConnectionPoolName=OraclePool;_RequestID=48f42491-030f-4306-bf31-6e0682769d18;|A potential connection leak detected for connection pool OraclePool. The stack trace of the thread is provided below :
com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:316)
com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:706)
com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:624)
com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:470)
com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:339)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:245)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:175)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:168)
com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
...
... application specific portion of stack omitted for brevity
...


In the Java thread dump the blocked request threads will look similar to the following thread stack:

"httpSSLWorkerThread-50001-127" daemon prio=3 tid=0x119e1000 nid=0x8eb4 waiting for monitor entry [0x60efb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:788)
    - waiting to lock <0x85225f90> (a com.sun.enterprise.resource.SJSASResourcePool)
    at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:682)
    at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:624)
    at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:470)
    at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
    at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:349)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:199)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:175)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:168)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
    ...
    ... application specific portion of stack omitted for brevity
    ...


In the above output, you can see that the thread is trying to lock object <0x85225f90>, which is an instance of SJSASResourcePool, which extends the AbstractResourcePool class. 

Searching the rest of the Java thread dump for a thread that has this object locked, you should find a TimerThread which is trying to close the physical JDBC connection from the method com.sun.gjc.spi.ManagedConnection.destroy()

"Timer-5" daemon prio=3 tid=0x035c8c00 nid=0xc0 waiting for monitor entry [0x6ccff000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:1179)
    - waiting to lock <0x95e4ec98> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.pool.OraclePooledConnection.close(OraclePooledConnection.java:247)
    - locked <0x95e4efd8> (a oracle.jdbc.driver.T4CXAConnection)
    at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:276)
    at com.sun.enterprise.resource.ConnectorAllocator.destroyResource(ConnectorAllocator.java:175)
    at com.sun.enterprise.resource.AbstractResourcePool.destroyResource(AbstractResourcePool.java:1001)
    at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:371)
    - locked <0x85225f90> (a com.sun.enterprise.resource.SJSASResourcePool)
    at com.sun.enterprise.resource.AbstractResourcePool.access$100(AbstractResourcePool.java:64)
    at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1688)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)


The thread name reporting the potential leak, "Timer-5", and the blocked thread are the same.

Changes

For this problem to be encountered you need to have:

  1. Enabled connection leak detection in a JDBC connection pool.
  2. Enabled connection leak reclamation in the same JDBC connection pool.
  3. Have a thread executing a SQL statement created from the connection at the point the application has had the connection for longer than the leak detection timeout.  Potentially the SQL statement could be running much more slowly than normal, perhaps due to performance issues on the database that its running on.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms