Connection Leaks From Wsh.servlet.RequestRates

(Doc ID 2366162.1)

Last updated on FEBRUARY 26, 2018

Applies to:

Oracle Shipping Execution - Version 12.2.5 and later
Information in this document applies to any platform.

Symptoms

Experiencing an issue in 12.2 EBS environment. Users reporting the system was very slow and/or they could not log in.
In the admin console there were approx 5,000 leaked connections. Tried to bounce apache and oacore but oacore was unresponsive.
Ended up killing the pids for the two oacore processes, then bouncing apache with it.
Additionally we bounced the admin server since oacore was not responding to abort commands.
This reset the leaked connection but we need to understand what happened and how to prevent it in the future.
 

Per the log file, Connections are leaking from wsh.servlet.RequestRates:

Call stack for leaked connection shows :

#### <oacore_server1> <[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:366)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:373)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:339)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:469)
at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:553)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513)
at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.apps.fnd.security.DBConnObjWLSDSPool.getConnection(DBConnObjWLSDSPool.java:373)
at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.dbConnect(AppsConnectionManagerWLSDS.java:223)
at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.localAppsConnect(AppsConnectionManagerWLSDS.java:193)
at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.makeGuestConnection(AppsConnectionManagerWLSDS.java:68)
at oracle.apps.fnd.security.DBConnObj.getLabelledConnection(DBConnObj.java:314)
at oracle.apps.fnd.security.DBConnObj.(DBConnObj.java:233)
at oracle.apps.fnd.security.DBConnObjDS.(DBConnObjDS.java:98)
at sun.reflect.GeneratedConstructorAccessor169.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at oracle.apps.fnd.common.Pool.createObject(Pool.java:1276)
at oracle.apps.fnd.security.DBConnObjWLSDSPool.borrowObjectX(DBConnObjWLSDSPool.java:173)
at oracle.apps.fnd.security.DBConnObjPool.borrowObject(DBConnObjPool.java:597)
at oracle.apps.fnd.security.AppsConnectionManager.borrowConnection(AppsConnectionManager.java:354)
at oracle.apps.fnd.common.Context.borrowConnection(Context.java:1758)
at oracle.apps.fnd.common.AppsContext.getPrivateConnectionFinal(AppsContext.java:2425)
at oracle.apps.fnd.common.AppsContext.getPrivateConnection(AppsContext.java:2362)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2218)
at oracle.apps.fnd.common.AppsContext.getLocalJDBCConnection(AppsContext.java:2601)
at oracle.apps.fnd.common.AppsContext.getLocalJDBCConnection(AppsContext.java:2536)
at oracle.apps.fnd.common.AppsContext.getLocalJDBCConnection(AppsContext.java:2517)
at oracle.apps.fnd.common.WebAppsContext.updateIcxSessionContext(WebAppsContext.java:3976)
at oracle.apps.fnd.common.WebAppsContext.setSecurityContext(WebAppsContext.java:4112)
at oracle.apps.fnd.common.WebAppsContext.updateSessionContext(WebAppsContext.java:3936)
at oracle.apps.wsh.servlet.IntegrationUtil.init(IntegrationUtil.java:135)
at oracle.apps.wsh.servlet.RequestRates.verifyCaller(RequestRates.java:181)
at oracle.apps.wsh.servlet.RequestRates.doGet(RequestRates.java:87)
at oracle.apps.wsh.servlet.RequestRates.doPost(RequestRates.java:144)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)


wsh.servlet.RequestRates calls IntegrationUtil to get connection but failed to return connection back to the pool



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms