STUCK Thread With FAT Lock When Closing Connection Causes Rest of Threads to Wait For New Connection (Doc ID 2215887.1)

Last updated on AUGUST 08, 2017

Applies to:

Oracle WebLogic Server - Version 9.2 and later
Information in this document applies to any platform.

Symptoms

On : WLS 10.3.6.0.13 PSU version, Oracle DB General component. This does not appear to be a problem specific to this PSU version.

STUCK Thread with FAT lock when closing connection causes rest of threads to wait for new connection

In customers production environment, during heavy load conditions, Stuck threads are seen with Fat Lock when closing the JDBC connections which is causing all the other threads blocked waiting for the new JDBC connection.

Environment: WLS 10.3.6.0.13

Stuck Threads from logs:

"[STUCK] ExecuteThread: '255' for queue: 'weblogic.kernel.Default (self-tuning)'" id=44546252 idx=0x2760 tid=5280 prio=1 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: weblogic/jdbc/common/internal/GenericConnectionPool@0x1135e4a90[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at weblogic/common/resourcepool/ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:394)[optimized]
at weblogic/common/resourcepool/ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:352)[optimized]
at weblogic/common/resourcepool/ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:339)[inlined]
at weblogic/jdbc/common/internal/ConnectionPool.reserve(ConnectionPool.java:469)[optimized]
at weblogic/jdbc/common/internal/ConnectionPool.reserve(ConnectionPool.java:368)[inlined]
at weblogic/jdbc/common/internal/ConnectionPool.reserve(ConnectionPool.java:353)[inlined]
at weblogic/jdbc/common/internal/MultiPool.searchLoadBalance(MultiPool.java:321)[optimized]
at weblogic/jdbc/common/internal/MultiPool.findPool(MultiPool.java:181)[optimized]
at weblogic/jdbc/common/internal/ConnectionPoolManager.reserve(ConnectionPoolManager.java:120)[inlined]
at weblogic/jdbc/common/internal/ConnectionPoolManager.reserve(ConnectionPoolManager.java:77)[inlined]
at weblogic/jdbc/jta/DataSource.getXAConnectionFromPool(DataSource.java:1791)[optimized]
at weblogic/jdbc/jta/DataSource.refreshXAConnAndEnlist(DataSource.java:1582)[inlined]
at weblogic/jdbc/jta/DataSource.getConnectionInternal(DataSource.java:499)[inlined]
at weblogic/jdbc/jta/DataSource.getConnection(DataSource.java:483)[optimized]
at weblogic/jdbc/common/internal/RmiDataSource.getConnectionInternal(RmiDataSource.java:527)[optimized]
at weblogic/jdbc/common/internal/RmiDataSource.getConnection(RmiDataSource.java:513)[inlined]
at weblogic/jdbc/common/internal/RmiDataSource.getConnection(RmiDataSource.java:506)[optimized]
at com/gxs/ics/j2ee/db/rac/RACConnection.refreshConnection(RACConnection.java:89)[optimized]
at com/gxs/ics/j2ee/db/rac/RACConnection.<init>(RACConnection.java:48)[inlined]
at com/gxs/ics/j2ee/db/DBConnection.getConnection(DBConnection.java:163)[inlined]
at com/gxs/ics/j2ee/db/DBConnection.getConnection(DBConnection.java:141)[inlined]
at com/gxs/ics/j2ee/db/DBConnection.getConnection(DBConnection.java:134)[inlined]
at com/gxs/ics/j2ee/db/MailSlot.select(MailSlot.java:47)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceImpl.authenticateUser(ICSServiceImpl.java:868)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceEJB.authenticateUser(Ljava/lang/String;Ljava/lang/String;C)Lcom/gxs/ics/valueobjects/ICSUserSessionInterface;(ICSServiceEJB.java)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServices_apy2wa_EOImpl.__WL_invoke(Ljava/lang/Object;[Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/ejb/container/internal/SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)[inlined]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServices_apy2wa_EOImpl.authenticateUser(Ljava/lang/String;Ljava/lang/String;C)Lcom/gxs/ics/valueobjects/ICSUserSessionInterface;(Unknown Source)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServices_apy2wa_EOImpl_WLSkel.invoke(ILweblogic/rmi/spi/InboundRequest;Lweblogic/rmi/spi/OutboundResponse;Ljava/lang/Object;)Lweblogic/rmi/spi/OutboundResponse;(Unknown Source)
at weblogic/rmi/internal/BasicServerRef.invoke(BasicServerRef.java:667)[optimized]
at weblogic/rmi/cluster/ClusterableServerRef.invoke(ClusterableServerRef.java:230)[optimized]
at weblogic/rmi/internal/BasicServerRef$1.run(BasicServerRef.java:522)[inlined]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)[inlined]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:146)[inlined]
at weblogic/rmi/internal/BasicServerRef.handleRequest(BasicServerRef.java:518)[inlined]
at weblogic/rmi/internal/wls/WLSExecuteRequest.run(WLSExecuteRequest.java:118)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:263)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace

 
"[STUCK] ExecuteThread: '136' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1594474 idx=0x2be4 tid=23483 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:300)[optimized]
at oracle/net/ns/DataPacket.receive(DataPacket.java:106)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:315)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:260)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:185)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:102)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[inlined]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readLongMSB(T4CSocketInputStreamWrapper.java:199)[optimized]
at oracle/jdbc/driver/T4CMAREngine.buffer2Value(T4CMAREngine.java:2374)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB8(T4CMAREngine.java:1358)[inlined]
at oracle/jdbc/driver/T4CTTIfun.processEOCS(T4CTTIfun.java:502)[inlined]
at oracle/jdbc/driver/T4CTTIfun.receive(T4CTTIfun.java:445)[optimized]
at oracle/jdbc/driver/T4CTTIfun.doRPC(T4CTTIfun.java:192)[optimized]
at oracle/jdbc/driver/T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)
at oracle/jdbc/driver/T4CConnection.logoff(T4CConnection.java:543)
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x13e590d80[recursive]
at oracle/jdbc/driver/PhysicalConnection.close(PhysicalConnection.java:3984)
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x13e590d80[thin lock]
at oracle/jdbc/pool/OraclePooledConnection.close(OraclePooledConnection.java:254)
^-- Holding lock: oracle/jdbc/driver/T4CXAConnection@0x13e5edb08[thin lock]
at weblogic/jdbc/wrapper/XAConnection.close(XAConnection.java:579)
at weblogic/jdbc/common/internal/ConnectionEnv.destroyForFlush(ConnectionEnv.java:618)
^-- Holding lock: weblogic/jdbc/common/internal/ConnectionEnv@0x113602ee0[recursive]
at weblogic/jdbc/common/internal/ConnectionEnv.destroy(ConnectionEnv.java:580)
^-- Holding lock: weblogic/jdbc/common/internal/ConnectionEnv@0x113602ee0[recursive]
at weblogic/jdbc/common/internal/ConnectionEnv.destroy(ConnectionEnv.java:562)
^-- Holding lock: weblogic/jdbc/common/internal/ConnectionEnv@0x113602ee0[biased lock]
at weblogic/common/resourcepool/ResourcePoolImpl.destroyResource(ResourcePoolImpl.java:1887)
at weblogic/common/resourcepool/ResourcePoolImpl.releaseResource(ResourcePoolImpl.java:647)[optimized]
^-- Holding lock: weblogic/jdbc/common/internal/GenericConnectionPool@0x1135e4a90[fat lock]
at weblogic/jdbc/common/internal/ConnectionPool.release(ConnectionPool.java:780)[optimized]
at weblogic/jdbc/common/internal/ConnectionPoolManager.release(ConnectionPoolManager.java:177)[optimized]
at weblogic/jdbc/wrapper/XAConnection.releaseToPool(XAConnection.java:422)[inlined]
at weblogic/jdbc/wrapper/JTAConnection.releaseXAConnIfNeeded(JTAConnection.java:270)[optimized]
^-- Holding lock: weblogic/jdbc/wrapper/JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection@0x1b2cc8570[biased lock]
at weblogic/jdbc/wrapper/JTAConnection.doClose(JTAConnection.java:644)[inlined]
at weblogic/jdbc/wrapper/JTAConnection.close(JTAConnection.java:455)[optimized]
at com/gxs/ics/j2ee/db/rac/RACConnection.close(RACConnection.java:143)[optimized]
at com/gxs/ics/j2ee/db/DBConnection.freeConnection(DBConnection.java:188)[optimized]
at com/gxs/ics/j2ee/db/XLogTable.select(XLogTable.java:221)[optimized]
at com/gxs/ics/j2ee/icssvcs/es/Send.delegatedSend(Send.java:1280)
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceImpl.delegatedSend(ICSServiceImpl.java:755)[inlined]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceEJB.delegatedSend(ICSServiceEJB.java:114)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceStreamingImpl.send(ICSServiceStreamingImpl.java:678)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceStreaming_snnlz1_EOImpl.__WL_invoke(Ljava/lang/Object;[Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/ejb/container/internal/SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)[inlined]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceStreaming_snnlz1_EOImpl.send(Lcom/gxs/ics/valueobjects/ICSUserSessionInterface;)Lcom/gxs/ics/valueobjects/ICSServiceResult;(Unknown Source)[optimized]
at com/gxs/ics/j2ee/ejb/icssvcs/ICSServiceStreaming_snnlz1_EOImpl_WLSkel.invoke(ILweblogic/rmi/spi/InboundRequest;Lweblogic/rmi/spi/OutboundResponse;Ljava/lang/Object;)Lweblogic/rmi/spi/OutboundResponse;(Unknown Source)[optimized]
at weblogic/rmi/internal/activation/ActivatableServerRef.invoke(ActivatableServerRef.java:85)[optimized]
at weblogic/rmi/internal/BasicServerRef$1.run(BasicServerRef.java:522)[inlined]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)[inlined]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:146)[inlined]
at weblogic/rmi/internal/BasicServerRef.handleRequest(BasicServerRef.java:518)[inlined]
at weblogic/rmi/internal/wls/WLSExecuteRequest.run(WLSExecuteRequest.java:118)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:263)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace


Steps to replicate issue:
===============

1) Create a WLS 10.3.6.0.13 domain with Admin and Managed servers which are part of the cluster.
2) Create a DataSource using Oracle DB.
3) Create JMS Server by using JDBC Store, UDQ.
4) Deploy the MDB TestMDB along with the TestThead class file which you attached to the file
5) Create the table in DB using test_table.sql
6) Start the MDB by sending a text message with 6 in as the text body to com.test queue
7) Add columns to the table using the columnadd.sql

Environment : WLS 10.3.6.0.13, DBVersion - Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

THis has not been found to be specific to these versions.

 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms