wm/SOAWorkManager is waiting to acquire lock 'oracle.mds.internal.cache.coherence.CoherenceCache' (Doc ID 2157858.1)

Last updated on JULY 06, 2016

Applies to:

Oracle Metadata Services - Version 12.1.3.0.0 to 12.2.1.0.0 [Release 12c]
Information in this document applies to any platform.

Symptoms

A SOA managed server experiences a "deadlocked thread" error  several times a day causing the managed server to crash.

####<06-Jun-2016 11:53:01 o'clock BST>
<Critical>
<Health>
<LW-ICT-SOA-02>
<ichnt_soa1>
<[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <734712cf-1698-41ae-8ecc-d67d3778fcb7-002e3550> <1465210381642>
<BEA-310006>
<Critical subsystem core has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>

 

 

The managed server log file will also show this deadlock thread dump:

[deadlocked thread] DaemonWorkThread: '176' of WorkManager: 'wm/SOAWorkManager':
-------------------------------------------------------------------------------
Thread 'DaemonWorkThread: '176' of WorkManager: 'wm/SOAWorkManager'' is waiting to acquire lock 'oracle.mds.core.CachedMOContent@16b7b500' that is held by thread 'DaemonWorkThread: '158' of WorkManager: 'wm/SOAWorkManager''

Stack trace:
------------
oracle.mds.core.CachedMOContent.setDocumentContent(CachedMOContent.java:698)
oracle.mds.core.MOContent.clearContent(MOContent.java:253)
oracle.mds.core.MOValueProvider.clearValue(MOValueProvider.java:305)
oracle.mds.internal.cache.CacheNode.clear(CacheNode.java:140)
oracle.mds.internal.cache.CacheNode.clear(CacheNode.java:175)
oracle.mds.internal.cache.CacheNode.clear(CacheNode.java:111)
oracle.mds.internal.cache.LayeredCache$MyCacheListener.objectInvalidated(LayeredCache.java:1552)
oracle.mds.internal.cache.coherence.CoherenceCache$CacheMapListener.handleEvent(CoherenceCache.java:740)
oracle.mds.internal.cache.coherence.CoherenceCache$CacheMapListener.entryDeleted(CoherenceCache.java:676)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:274)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:573)
com.tangosol.coherence.component.util.SafeNamedCache.translateMapEvent(SafeNamedCache.CDB:7)
com.tangosol.coherence.component.util.SafeNamedCache.entryDeleted(SafeNamedCache.CDB:1)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:274)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:573)
com.tangosol.coherence.component.util.LocalCache$CacheHandler.translateMapEvent(LocalCache.CDB:5)
com.tangosol.coherence.component.util.LocalCache$CacheHandler.entryDeleted(LocalCache.CDB:1)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:274)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:573)
com.tangosol.net.cache.OldCache.dispatchEvent(OldCache.java:1980)
com.tangosol.net.cache.OldCache$Entry.discard(OldCache.java:2400)
com.tangosol.net.cache.OldCache.removeEvicted(OldCache.java:1594)
com.tangosol.net.cache.LocalCache.removeEvicted(LocalCache.java:548)
com.tangosol.net.cache.OldCache.pruneIncremental(OldCache.java:1906)
com.tangosol.net.cache.OldCache.prune(OldCache.java:1683)
com.tangosol.net.cache.OldCache.put(OldCache.java:289)
com.tangosol.net.cache.OldCache.put(OldCache.java:230)
com.tangosol.util.WrapperObservableMap.put(WrapperObservableMap.java:151)
com.tangosol.util.WrapperConcurrentMap.put(WrapperConcurrentMap.java:273)
com.tangosol.coherence.component.util.jb_ConcurrentMap.super$put(ConcurrentMap.CDB:1)
com.tangosol.coherence.component.util.sink_ConcurrentMap.put(ConcurrentMap.CDB:1)
com.tangosol.coherence.component.util.ConcurrentMap.put(ConcurrentMap.CDB:1)
com.tangosol.coherence.component.util.SafeNamedCache.put(SafeNamedCache.CDB:1)
oracle.mds.internal.cache.coherence.CoherenceCache.put(CoherenceCache.java:217)
oracle.mds.core.CachedMOContent.cacheData(CachedMOContent.java:233)
oracle.mds.core.CachedMOContent.fetchContent(CachedMOContent.java:315)
oracle.mds.core.MOContent.getMElemBean(MOContent.java:747)
oracle.mds.core.MetadataObject.getContentWithRetry(MetadataObject.java:3136)
oracle.mds.core.BeanMetadataObject.getBean(BeanMetadataObject.java:88)
oracle.mds.melement.MElementBeanImpl.getReferencedBean(MElementBeanImpl.java:1197)
oracle.mds.melement.MElementBeanImpl._getReferencedBean(MElementBeanImpl.java:633)
oracle.mds.melement.MElementBeanImpl._getReferencedBeanWithIntegrity(MElementBeanImpl.java:663)
oracle.tip.b2b.model.profile.AgreementTypeImpl$SupportedDocumentTypeTypeImpl.getDocumentDefinition(AgreementTypeImpl.java:396)
oracle.tip.b2b.tpa.RepoDataAccessor.addDocumentInfo(RepoDataAccessor.java:1450)
oracle.tip.b2b.tpa.RepoDataAccessor.getAgreementDetails(RepoDataAccessor.java:564)
oracle.tip.b2b.tpa.TPAProcessor.processTPA(TPAProcessor.java:555)
oracle.tip.b2b.tpa.TPAProcessor.processOutgoingTPA(TPAProcessor.java:227)
oracle.tip.b2b.engine.Engine.processOutgoingMessageImpl(Engine.java:1614)
oracle.tip.b2b.engine.Engine.processOutgoingMessage(Engine.java:907)
oracle.tip.b2b.engine.Engine.outgoingContinueProcess(Engine.java:4168)
oracle.tip.b2b.engine.Engine.handleMessageEvent(Engine.java:3975)
oracle.tip.b2b.engine.Engine.processEvents(Engine.java:3499)
oracle.tip.b2b.engine.ThreadWorkExecutor.processEvent(ThreadWorkExecutor.java:696)
oracle.tip.b2b.engine.ThreadWorkExecutor.run(ThreadWorkExecutor.java:223)
oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)
weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

[deadlocked thread] DaemonWorkThread: '158' of WorkManager: 'wm/SOAWorkManager':
-------------------------------------------------------------------------------
Thread 'DaemonWorkThread: '158' of WorkManager: 'wm/SOAWorkManager'' is waiting to acquire lock 'oracle.mds.internal.cache.coherence.CoherenceCache@4b4aaf07' that is held by thread 'DaemonWorkThread: '176' of WorkManager: 'wm/SOAWorkManager''

Stack trace:
------------
oracle.mds.internal.cache.coherence.CoherenceCache.put(CoherenceCache.java:201)
oracle.mds.core.CachedMOContent.cacheData(CachedMOContent.java:233)
oracle.mds.core.CachedMOContent.fetchContent(CachedMOContent.java:315)
oracle.mds.core.MOContent.getMElemBean(MOContent.java:747)
oracle.mds.core.MetadataObject.getContentWithRetry(MetadataObject.java:3136)
oracle.mds.core.BeanMetadataObject.getBean(BeanMetadataObject.java:88)
oracle.mds.melement.MElementBeanImpl.getReferencedBean(MElementBeanImpl.java:1197)
oracle.mds.melement.MElementBeanImpl._getReferencedBean(MElementBeanImpl.java:633)
oracle.mds.melement.MElementBeanImpl._getReferencedBeanWithIntegrity(MElementBeanImpl.java:663)
oracle.tip.b2b.model.profile.AgreementTypeImpl$SupportedDocumentTypeTypeImpl.getDocumentDefinition(AgreementTypeImpl.java:396)
oracle.tip.b2b.tpa.RepoDataAccessor.addDocumentInfo(RepoDataAccessor.java:1450)
oracle.tip.b2b.tpa.RepoDataAccessor.getAgreementDetails(RepoDataAccessor.java:564)
oracle.tip.b2b.tpa.TPAProcessor.processTPA(TPAProcessor.java:555)
oracle.tip.b2b.tpa.TPAProcessor.processOutgoingTPA(TPAProcessor.java:227)
oracle.tip.b2b.engine.Engine.processOutgoingMessageImpl(Engine.java:1614)
oracle.tip.b2b.engine.Engine.processOutgoingMessage(Engine.java:907)
oracle.tip.b2b.engine.Engine.outgoingContinueProcess(Engine.java:4168)
oracle.tip.b2b.engine.Engine.handleMessageEvent(Engine.java:3975)
oracle.tip.b2b.engine.Engine.processEvents(Engine.java:3499)
oracle.tip.b2b.engine.ThreadWorkExecutor.processEvent(ThreadWorkExecutor.java:696)
oracle.tip.b2b.engine.ThreadWorkExecutor.run(ThreadWorkExecutor.java:223)
oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)
weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)

. SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Critical THREAD = [ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000394 MACHINE = LW-ICT-SOA-02 TXID = CONTEXTID = b4cdd371-c287-4e89-bd4b-22b40d09faaa-0048286e TIMESTAMP = 1465322440888
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000

 

 


The managed server is restarted successfully but the error happens again shortly after.


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms