WebLogic Server (WLS) Hangs With Informix JDBC Driver Timer (statement-timeout) and Poll() Threads on AIX (Doc ID 1437576.1)

Last updated on JUNE 08, 2012

Applies to:

Oracle Weblogic Server - Version 8.1 to 10.3
IBM AIX on POWER Systems (64-bit)
IBM AIX on POWER Systems (32-bit)

Symptoms

Customer application works smoothly at the first hour of restart WebLogic server, but will always hangs later for almost all database activities. Even on simple test-unused event for data source, for example:

"[STUCK] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x000000012C40D200, sys_thread_t:0x000000012A87F3B8, state:CW, native ID:0x00000000035500CD) prio=1
at com/informix/jdbc/IfxStatement.a(IfxStatement.java:1322(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.d(IfxPreparedStatement.java:443(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.execute(IfxPreparedStatement.java:428(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionEnv.test(ConnectionEnv.java:718(Compiled Code))
at weblogic/jdbc/common/internal/ConnectionEnv.test(ConnectionEnv.java:460(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1504(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1413(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1401(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:1806(Compiled Code))
at weblogic/common/resourcepool/ResourcePoolImpl.access$700(ResourcePoolImpl.java:37)
at weblogic/common/resourcepool/ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2002(Compiled Code))
at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:265(Compiled Code))
at weblogic/work/ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518(Compiled Code))
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209(Compiled Code))
at weblogic/work/ExecuteThread.run(ExecuteThread.java:181(Compiled Code))


Typical stack trace is Informix JDBC driver try to cancel current statement:

VM thread list lock (0x0000000110016A30): Flat locked by "Timer-198407" (0x000000012C1D7400), entry count 1
Waiting to enter:
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x0000000117B60E00)
"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x0000000123F81A00)
"[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012B52B700)
"[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012B754B00)
"[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012B9E5000)
"[STUCK] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012B515300)
"[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x0000000124366F00)
"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x00000001298FF600)
"Timer-197550" (0x000000012BD7A400)
"Timer-198323" (0x000000012987CF00)
"Timer-198338" (0x000000012BCF7B00)
"Timer-198373" (0x000000012C2C7D00)
"Timer-198415" (0x000000012B439B00)
"Timer-198312" (0x00000001286D8600)
"Timer-198317" (0x0000000129A3E200)
"Timer-198316" (0x000000012A600400)
"Timer-198387" (0x000000012A6AAD00)
"Timer-198384" (0x000000012C1D8500)
"Timer-198420" (0x000000012C1DAD00)
"Timer-198397" (0x000000012C1DE600)
"Timer-198427" (0x00000001296F1500)
"Timer-198428" (0x0000000129506900)

"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x0000000117B60E00, sys_thread_t:0x0000000116E83850, state:B, native ID:0x00000000025500FD) prio=1
at java/lang/Thread.startImpl(Native Method)
at java/lang/Thread.start(Thread.java:980(Compiled Code))
at java/util/Timer.<init>(Timer.java:179(Compiled Code))
at java/util/Timer.<init>(Timer.java:147(Compiled Code))
at com/informix/jdbc/IfxCancelQueryImpl.startCancel(IfxCancelQueryImpl.java:37(Compiled Code))
at com/informix/jdbc/IfxStatement.executeQueryImpl(IfxStatement.java:1217(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:376(Compiled Code))
at weblogic/jdbc/wrapper/PreparedStatement.executeQuery(PreparedStatement.java:100(Compiled Code))
at com/p6spy/engine/logging/P6LogPreparedStatement.executeQuery(P6LogPreparedStatement.java:172(Compiled Code))
at org/hibernate/jdbc/AbstractBatcher.getResultSet(AbstractBatcher.java:208(Compiled Code))
at org/hibernate/loader/Loader.getResultSet(Loader.java:1812(Compiled Code))
at org/hibernate/loader/Loader.doQuery(Loader.java:697(Compiled Code))

"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x000000011B72F300, sys_thread_t:0x000000011BD7E738, state:CW, native ID:0x00000000014600FD) prio=5
at com/informix/jdbc/IfxStatement.executeQueryImpl(IfxStatement.java:1229(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:376(Compiled Code))
at weblogic/jdbc/wrapper/PreparedStatement.executeQuery(PreparedStatement.java:100(Compiled Code))
at com/p6spy/engine/logging/P6LogPreparedStatement.executeQuery(P6LogPreparedStatement.java:172(Compiled Code))
at org/hibernate/jdbc/AbstractBatcher.getResultSet(AbstractBatcher.java:208(Compiled Code))
at org/hibernate/loader/Loader.getResultSet(Loader.java:1812(Compiled Code))
at org/hibernate/loader/Loader.doQuery(Loader.java:697(Compiled Code))

"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x0000000123F81A00, sys_thread_t:0x000000011DFE8FF0, state:B, native ID:0x0000000003BE009B) prio=1
at java/lang/Thread.startImpl(Native Method)
at java/lang/Thread.start(Thread.java:980(Compiled Code))
at java/util/Timer.<init>(Timer.java:179(Compiled Code))
at java/util/Timer.<init>(Timer.java:147(Compiled Code))
at com/informix/jdbc/IfxCancelQueryImpl.startCancel(IfxCancelQueryImpl.java:37(Compiled Code))
at com/informix/jdbc/IfxStatement.executeQueryImpl(IfxStatement.java:1217(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:376(Compiled Code))
at weblogic/jdbc/wrapper/PreparedStatement.executeQuery(PreparedStatement.java:100(Compiled Code))
at com/p6spy/engine/logging/P6LogPreparedStatement.executeQuery(P6LogPreparedStatement.java:172(Compiled Code))
at org/hibernate/jdbc/AbstractBatcher.getResultSet(AbstractBatcher.java:208(Compiled Code))
at org/hibernate/loader/Loader.getResultSet(Loader.java:1812(Compiled Code))
at org/hibernate/loader/Loader.doQuery(Loader.java:697(Compiled Code))

3XMTHREADINFO "ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'" (TID:0x0000000118634900, sys_thread_t:0x000000011840B7F0, state:R, native ID:0x000000000258007B) prio=5
4XESTACKTRACE at weblogic/socket/PosixSocketMuxer.poll(Native Method)
4XESTACKTRACE at weblogic/socket/PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102)
4XESTACKTRACE at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
4XESTACKTRACE at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
4XESTACKTRACE at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
4XESTACKTRACE at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)


Another example is the Timer thread try to stop cancel query statement:

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x000000012B48AD00, sys_thread_t:0x00000001290DA818, state:CW, native ID:0x000000000378006D) prio=5
at java/util/Timer.cancel(Timer.java:411(Compiled Code))
at com/informix/jdbc/IfxCancelQueryImpl.stopCancel(IfxCancelQueryImpl.java:43(Compiled Code))
at com/informix/jdbc/IfxStatement.executeQueryImpl(IfxStatement.java:1209(Compiled Code))
at com/informix/jdbc/IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:376(Compiled Code))
at weblogic/jdbc/wrapper/PreparedStatement.executeQuery(PreparedStatement.java:100(Compiled Code))
at com/p6spy/engine/logging/P6LogPreparedStatement.executeQuery(P6LogPreparedStatement.java:172(Compiled Code))
at org/hibernate/jdbc/AbstractBatcher.getResultSet(AbstractBatcher.java:208(Compiled Code))
at org/hibernate/loader/Loader.getResultSet(Loader.java:1812(Compiled Code))
at org/hibernate/loader/Loader.doQuery(Loader.java:697(Compiled Code))
at org/hibernate/loader/Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259(Compiled Code))
at org/hibernate/loader/Loader.doList(Loader.java:2232(Compiled Code))

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms