WebCenter Portal Hangs Frequently with Stuck Threads in oracle.adf.model.dcframe.DataControlFrameImpl.lock and oracle.portlet.client.service.pipeline.FutureConditionTaskWithTimeout.getWithTimeout (Doc ID 1673551.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle WebCenter Portal - Version 11.1.1.7.0 and later
Information in this document applies to any platform.

Symptoms

Custom Portal running in production mainly uses WSRP2 portlets. There are 4 managed servers running in a cluster.
Both the Portal and the portlets are deployed into the same managed server called WC_Portlet, but the managed server has both the Portlet and the Portal libraries needed.

The website intermittently slows down significantly. When that happens then the Managed Server(s) running the Portal become into a Warning state, and the cluster members drop out.
When the site runs normally then the homepage renders in 4-6 seconds. When it slows down then the response tim becomes 15-20 seconds, but sometimes even 1-2 minutes.
When the issue happens there are stuck threads seen with the following stack:

"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.locks.ReentrantLock$NonfairSync@36e636e6 WAITING native
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:171)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
oracle.adf.model.dcframe.DataControlFrameImpl.lock(DataControlFrameImpl.java:490)
oracle.adf.model.dcframe.DataControlFrameImpl.beginRequest(DataControlFrameImpl.java:581)
oracle.adf.model.BindingContext.setCurrentFrame(BindingContext.java:2374)
oracle.adf.model.BindingContext.setCurrentDataControlFrame(BindingContext.java:2258)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


"[STUCK] ExecuteThread: '366' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.concurrent.FutureTask$Sync@20282028 TIMED_WAITING native
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:211)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
java.util.concurrent.FutureTask.get(FutureTask.java:91)
oracle.portlet.client.service.pipeline.FutureConditionTaskWithTimeout.getWithTimeout(FutureConditionTaskWithTimeout.java:54)
oracle.portlet.client.tech.common.PortletFutureUtils.getInternal(PortletFutureUtils.java:87)
oracle.portlet.client.tech.common.PortletFutureUtils.get(PortletFutureUtils.java:35)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.performBlockingInteraction(WSRPClientImpl.java:1030)
oracle.portlet.client.techimpl.wsrp.WSRPClientImpl.processAction(WSRPClientImpl.java:903)
oracle.portlet.client.containerimpl.wsrp.WSRPRunnablePortletImpl._processAction(WSRPRunnablePortletImpl.java:999)
...
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


"[STUCK] ExecuteThread: '359' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.rjvm.ResponseImpl@58745874 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:90)
weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:130)
weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:110)
weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
weblogic.cluster.replication.ReplicationManager_1036_WLStub.update(Unknown Source)
sun.reflect.GeneratedMethodAccessor1308.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:600)
weblogic.cluster.replication.SecureReplicationInvocationHandler$ReplicationServicesInvocationAction.run(SecureReplicationInvocationHandler.java:194)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.cluster.replication.SecureReplicationInvocationHandler.invoke(SecureReplicationInvocationHandler.java:164)
com.sun.proxy.$Proxy191.update(Unknown Source)
weblogic.cluster.replication.ReplicationManager.sendUpdateRequestToSecondary(ReplicationManager.java:740)
weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:662)
weblogic.servlet.internal.session.ReplicatedSessionData.syncSession(ReplicatedSessionData.java:666)
weblogic.servlet.internal.session.ReplicatedSessionContext.sync(ReplicatedSessionContext.java:83)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2943)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.syncSession(ServletRequestImpl.java:2918)
weblogic.servlet.internal.ServletResponseImpl$1.run(ServletResponseImpl.java:1504)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1498)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


The issue goes away automatically after 20-30 minutes, or after a restart of the servers.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms