Stuck Threads Exalogic Server- A Java.util.concurrent.locks.ReentrantReadWriteLock$FairSync (Doc ID 2294071.1)

Last updated on AUGUST 31, 2017

Applies to:

Oracle WebLogic Server - Version 12.2.1.1.0 and later
Information in this document applies to any platform.

Symptoms

On : 12.2.1.1.0 version, WLS Build

Stuck Threads Exalogic CUST Server- a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync

Lately in our Exalogic CUST Servers stuck threads appear randomly during the operation hours with the following stuck trace :

"[STUCK] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" #26130 daemon prio=1 os_prio=0 tid=0x00007f5c456b2800 nid=0x1dac waiting on condition [0x00007f5bedadf000]
  
  java.lang.Thread.State: WAITING (parking)
  
  at sun.misc.Unsafe.park(Native Method)
  
  - parking to wait for (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
  
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
  
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
  
  at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
  
  at oracle.dms.context.internal.AbstractContextFamily.lockCtxMapForWrite(AbstractContextFamily.java:329)
  
  at oracle.dms.context.internal.wls2.WLS2ContextFamily.addContext(WLS2ContextFamily.java:86)
  
  at oracle.dms.context.internal.wls2.WLS2ContextManager.createContext(WLS2ContextManager.java:148)
  
  at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:110)
  
  at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:45)
  
  at oracle.dms.context.internal.AbstractContextManager.getContext(AbstractContextManager.java:281)
  
.............


See the following error:

"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" #115 daemon prio=1 os_prio=0

tid=0x00007fd7ccb4c800 nid=0x610 waiting on condition [0x00007fd7b4619000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x0000000416ea6a08> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)

at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)

at oracle.dms.context.internal.AbstractContextFamily.lockCtxMapForWrite(AbstractContextFamily.java:329)

at oracle.dms.context.internal.wls2.WLS2ContextFamily.addContext(WLS2ContextFamily.java:86)

at oracle.dms.context.internal.wls2.WLS2ContextManager.createContext(WLS2ContextManager.java:148)

at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:110)

at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:45)

at oracle.dms.context.internal.AbstractContextManager.getContext(AbstractContextManager.java:281)

at oracle.dms.context.DMSContextManager.getDomainExecutionContext(DMSContextManager.java:1514)

at oracle.dms.context.ECForJDBC.get(ECForJDBC.java:276)

at oracle.dms.context.ECForJDBC.getMap(ECForJDBC.java:373)

at oracle.jdbc.driver.PhysicalConnection.updateSystemContext11(PhysicalConnection.java:6570)

at oracle.jdbc.driver.PhysicalConnection.updateSystemContext(PhysicalConnection.java:6486)

at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1201)

at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:5010)

at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:5136)

- locked <0x00000003f19b4780> (a oracle.jdbc.driver.T4CConnection)

at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1519)

at weblogic.jdbc.wrapper.PreparedStatement.executeUpdate(PreparedStatement.java:170)

at gr.ote.bus.log.beans.db.FalloutDBBean.insertinBUS_MONITOR_FAILED(FalloutDBBean.java:318)

at gr.ote.bus.log.beans.db.FalloutDBBean.onMessage(FalloutDBBean.java:210)

at sun.reflect.GeneratedMethodAccessor385.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)

at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint

(ReflectiveMethodInvocation.java:182)

at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed

(ReflectiveMethodInvocation.java:149)

at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)

at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101)

at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed

(ReflectiveMethodInvocation.java:171)

at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)

at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke

(AbstractEJBRequestScopeActivationInterceptor.java:73)

at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)

at sun.reflect.GeneratedMethodAccessor383.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)

at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed

(ReflectiveMethodInvocation.java:171)

at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed

(DelegatingIntroductionInterceptor.java:131)

at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke

(DelegatingIntroductionInterceptor.java:119)

at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed

(ReflectiveMethodInvocation.java:171)

at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)

at com.sun.proxy.$Proxy259.onMessage(Unknown Source)

at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)

at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)

at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:297)

at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:5107)

at weblogic.jms.client.JMSSession.execute(JMSSession.java:4775)

at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:4170)

at weblogic.jms.client.JMSSession.access$000(JMSSession.java:127)

at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5627)

at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:666)

at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)

at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)

at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)

at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)

 

"DFW Diagnostic Dump Sampling - JVMThreadDump" #88 daemon prio=5 os_prio=0 tid=0x00007fd7c02b4800 nid=0x59d waiting on condition

[0x00007fd7b8a98000]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x0000000416ea6a08> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)

at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)

at oracle.dms.context.internal.AbstractContextFamily.lockCtxMapForRead(AbstractContextFamily.java:345)

at oracle.dms.context.internal.AbstractContextFamily.getContexts(AbstractContextFamily.java:182)

at oracle.dms.context.internal.wls2.WLS2ContextManager.getThreadId2ContextMap(WLS2ContextManager.java:366)

at oracle.dms.context.DMSContextManager.getThreadId2ContextMap(DMSContextManager.java:626)

at oracle.dfw.impl.dump.ThreadDump.takeSnapshot(ThreadDump.java:708)

at oracle.dfw.impl.dump.ThreadDump.internalThreadDump(ThreadDump.java:507)

at oracle.dfw.impl.dump.ThreadDump.executeDump(ThreadDump.java:311)

at oracle.dfw.sampling.DumpSampling.executeDump(DumpSampling.java:615)

at oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:565)

- locked <0x0000000345fb4000> (a oracle.dfw.sampling.DumpSampling)

 

"oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Mon Apr 10 17:23:52 EEST 2017)" #1647

daemon prio=5 os_prio=0 tid=0x00007fd7fc14b800 nid=0x3897 waiting on condition [0x00007fd7b4415000]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x0000000416ea6a08> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)

at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)

at oracle.dms.context.internal.AbstractContextFamily.lockCtxMapForRead(AbstractContextFamily.java:345)

at oracle.dms.context.internal.AbstractContextFamily.getContexts(AbstractContextFamily.java:182)

at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:311)

at oracle.dms.context.internal.wls2.WLS2ContextManager.getContext(WLS2ContextManager.java:45)

at oracle.dms.context.DMSContextManager.getContext(DMSContextManager.java:592)

at oracle.dms.context.ExecutionContext.getActiveContext(ExecutionContext.java:143)

at oracle.dms.spy.jvm.ThreadMonitor.refresh(ThreadMonitor.java:393)

at oracle.dms.spy.jvm.ThreadMonitor.<init>(ThreadMonitor.java:246)

.........

 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms