WebCenter Portal Hangs Frequently with Stuck Threads in oracle.adf.model.dcframe.DataControlFrameImpl.lock and oracle.portlet.client.service.pipeline.FutureConditionTaskWithTimeout.getWithTimeout
(Doc ID 1673551.1)
Last updated on DECEMBER 19, 2022
Applies to:
Oracle WebCenter Portal - Version 11.1.1.7.0 to 11.1.1.9.190115 [Release 11g]Information in this document applies to any platform.
Symptoms
Custom Portal (Framework) running in production mainly uses WSRP2 portlets. There are 4 managed servers running in a cluster.
Both the Portal and the portlets are deployed into the same managed server called WC_Portlet, but the managed server has both the Portlet and the Portal libraries needed.
The website intermittently slows down significantly. When that happens then the Managed Server(s) running the Portal become into a Warning state, and the cluster members drop out.
When the site runs normally then the homepage renders in 4-6 seconds. When it slows down then the response tim becomes 15-20 seconds, but sometimes even 1-2 minutes.
When the issue happens there are stuck threads seen with the following stack:
"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.locks.ReentrantLock$NonfairSync@36e636e6 WAITING native
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:171)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
oracle.adf.model.dcframe.DataControlFrameImpl.lock(DataControlFrameImpl.java:490)
oracle.adf.model.dcframe.DataControlFrameImpl.beginRequest(DataControlFrameImpl.java:581)
oracle.adf.model.BindingContext.setCurrentFrame(BindingContext.java:2374)
oracle.adf.model.BindingContext.setCurrentDataControlFrame(BindingContext.java:2258)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
"[STUCK] ExecuteThread: '366' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.FutureTask$Sync@20282028 TIMED_WAITING native
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:211)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
java.util.concurrent.FutureTask.get(FutureTask.java:91)
oracle.portlet.client.service.pipeline.FutureConditionTaskWithTimeout.getWithTimeout(FutureConditionTaskWithTimeout.java:54)
oracle.portlet.client.tech.common.PortletFutureUtils.getInternal(PortletFutureUtils.java:87)
oracle.portlet.client.tech.common.PortletFutureUtils.get(PortletFutureUtils.java:35)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.performBlockingInteraction(WSRPClientImpl.java:1030)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.processAction(WSRPClientImpl.java:903)
oracle.portlet.client.containerimpl.wsrp.WSRPRunnablePortletImpl._processAction(WSRPRunnablePortletImpl.java:999)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
"[STUCK] ExecuteThread: '359' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.rjvm.ResponseImpl@58745874 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:90)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:130)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:110)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
weblogic.cluster.replication.ReplicationManager_1036_WLStub.update(Unknown Source)
sun.reflect.GeneratedMethodAccessor1308.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:600)
weblogic.cluster.replication.SecureReplicationInvocationHandler$ReplicationServicesInvocationAction.run(SecureReplicationInvocationHandler.java:194)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.cluster.replication.SecureReplicationInvocationHandler.invoke(SecureReplicationInvocationHandler.java:164)
com.sun.proxy.$Proxy191.update(Unknown Source)
weblogic.cluster.replication.ReplicationManager.sendUpdateRequestToSecondary(ReplicationManager.java:740)
weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:662)
weblogic.servlet.internal.session.ReplicatedSessionData.syncSession(ReplicatedSessionData.java:666)
weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:83)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2943)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2918)
weblogic.servlet.internal.ServletResponseImpl$1.run(ServletResponseImpl.java:1504)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1498)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:171)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
oracle.adf.model.dcframe.DataControlFrameImpl.lock(DataControlFrameImpl.java:490)
oracle.adf.model.dcframe.DataControlFrameImpl.beginRequest(DataControlFrameImpl.java:581)
oracle.adf.model.BindingContext.setCurrentFrame(BindingContext.java:2374)
oracle.adf.model.BindingContext.setCurrentDataControlFrame(BindingContext.java:2258)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
"[STUCK] ExecuteThread: '366' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.FutureTask$Sync@20282028 TIMED_WAITING native
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:211)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
java.util.concurrent.FutureTask.get(FutureTask.java:91)
oracle.portlet.client.service.pipeline.FutureConditionTaskWithTimeout.getWithTimeout(FutureConditionTaskWithTimeout.java:54)
oracle.portlet.client.tech.common.PortletFutureUtils.getInternal(PortletFutureUtils.java:87)
oracle.portlet.client.tech.common.PortletFutureUtils.get(PortletFutureUtils.java:35)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.performBlockingInteraction(WSRPClientImpl.java:1030)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.processAction(WSRPClientImpl.java:903)
oracle.portlet.client.containerimpl.wsrp.WSRPRunnablePortletImpl._processAction(WSRPRunnablePortletImpl.java:999)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
"[STUCK] ExecuteThread: '359' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.rjvm.ResponseImpl@58745874 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:90)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:130)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:110)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
weblogic.cluster.replication.ReplicationManager_1036_WLStub.update(Unknown Source)
sun.reflect.GeneratedMethodAccessor1308.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:600)
weblogic.cluster.replication.SecureReplicationInvocationHandler$ReplicationServicesInvocationAction.run(SecureReplicationInvocationHandler.java:194)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.cluster.replication.SecureReplicationInvocationHandler.invoke(SecureReplicationInvocationHandler.java:164)
com.sun.proxy.$Proxy191.update(Unknown Source)
weblogic.cluster.replication.ReplicationManager.sendUpdateRequestToSecondary(ReplicationManager.java:740)
weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:662)
weblogic.servlet.internal.session.ReplicatedSessionData.syncSession(ReplicatedSessionData.java:666)
weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:83)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2943)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2918)
weblogic.servlet.internal.ServletResponseImpl$1.run(ServletResponseImpl.java:1504)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1498)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
The issue goes away automatically after 20-30 minutes, or after a restart of the servers.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |