Sites Cluster Node Taking Too Long to Respond: Threads BLOCKING on net.sf.ehcache.distribution.ManualRMICacheManagerPeerProvider.listRemoteCachePeers() (Doc ID 2517942.1)

Last updated on MARCH 05, 2024

Applies to:

Symptoms

Sites Cluster Node Taking Too Long to Respond: Threads BLOCKING on net.sf.ehcache.distribution.ManualRMICacheManagerPeerProvider.listRemoteCachePeers()

One of the nodes in the Sites cluster is taking too long to respond.
- Several requests timed out due to this.
- This unresponsive node is also running the Admin server.

STUCK thread reports in managed server log indicate threads taking a long time invalidated cache dependencies:
e.g.
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "601" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@123456[GET /sites/Satellite]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
net.sf.ehcache.distribution.ManualRMICacheManagerPeerProvider.listRemoteCachePeers(ManualRMICacheManagerPeerProvider.java:82)
net.sf.ehcache.distribution.RMISynchronousCacheReplicator.listRemoteCachePeers(RMISynchronousCacheReplicator.java:335)
net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicateRemovalNotification(RMISynchronousCacheReplicator.java:239)
net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementRemoved(RMISynchronousCacheReplicator.java:229)
net.sf.ehcache.event.RegisteredEventListeners.notifyListener(RegisteredEventListeners.java:302)
net.sf.ehcache.event.RegisteredEventListeners.invokeListener(RegisteredEventListeners.java:284)
net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementRemoved(RegisteredEventListeners.java:108)
net.sf.ehcache.event.RegisteredEventListeners.notifyElementRemoved(RegisteredEventListeners.java:87)
net.sf.ehcache.Cache.notifyRemoveInternalListeners(Cache.java:2399)
net.sf.ehcache.Cache.removeInternal(Cache.java:2373)
net.sf.ehcache.Cache.remove(Cache.java:2283)
net.sf.ehcache.Cache.remove(Cache.java:2208)
net.sf.ehcache.Cache.remove(Cache.java:2186)
com.fatwire.cache.ehcache.EhCacheObjectCache.invalidateDependencies(EhCacheObjectCache.java:362)
com.fatwire.cache.ehcache.EhCacheObjectCache.invalidateDependencies(EhCacheObjectCache.java:346)
COM.FutureTense.Cache.LinkedCacheOverObjectCache.invalidateDependencies(LinkedCacheOverObjectCache.java:117)
COM.FutureTense.Common.FResultCacher.killCacheHash(FResultCacher.java:441)
  ...
Thread dumps show many BLOCKING threads while attempting to invalidate cache dependencies:
e.g.
java.lang.Thread.State: BLOCKED (on object monitor)
at net.sf.ehcache.distribution.ManualRMICacheManagerPeerProvider.listRemoteCachePeers(ManualRMICacheManagerPeerProvider.java:82)
- waiting to lock (a net.sf.ehcache.distribution.ManualRMICacheManagerPeerProvider)
at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.listRemoteCachePeers(RMISynchronousCacheReplicator.java:335)
at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.replicateRemovalNotification(RMISynchronousCacheReplicator.java:239)
at net.sf.ehcache.distribution.RMISynchronousCacheReplicator.notifyElementRemoved(RMISynchronousCacheReplicator.java:229)
at net.sf.ehcache.event.RegisteredEventListeners.notifyListener(RegisteredEventListeners.java:302)
at net.sf.ehcache.event.RegisteredEventListeners.invokeListener(RegisteredEventListeners.java:284)
at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementRemoved(RegisteredEventListeners.java:108)
at net.sf.ehcache.event.RegisteredEventListeners.notifyElementRemoved(RegisteredEventListeners.java:87)
at net.sf.ehcache.Cache.notifyRemoveInternalListeners(Cache.java:2399)
at net.sf.ehcache.Cache.removeInternal(Cache.java:2373)
at net.sf.ehcache.Cache.remove(Cache.java:2283)
at net.sf.ehcache.Cache.remove(Cache.java:2208)
at net.sf.ehcache.Cache.remove(Cache.java:2186)
at com.fatwire.cache.ehcache.EhCacheObjectCache.invalidateDependencies(EhCacheObjectCache.java:362)
at com.fatwire.cache.ehcache.EhCacheObjectCache.invalidateDependencies(EhCacheObjectCache.java:346)
at COM.FutureTense.Cache.LinkedCacheOverObjectCache.invalidateDependencies(LinkedCacheOverObjectCache.java:117)
at COM.FutureTense.Common.FResultCacher.killCacheHash(FResultCacher.java:441)

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.