EM 13.4 : OMS Crashed With Error "<BEA-000394> <Deadlock detected: [deadlocked thread] oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl"
(Doc ID 2717391.1)
Last updated on JANUARY 17, 2023
Applies to:
Enterprise Manager Base Platform - Version 13.4.1.0.0 and laterInformation in this document applies to any platform.
Symptoms
13.4 OMS got crashed and observed the below errors in oms logs:
$EM_INSTANCE_HOME/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS<n>.out:
<Sep 15, 2020 3:23:52,948 AM EDT> <Critical> <WebLogicServer> <BEA-000394> <Deadlock detected:
[deadlocked thread] oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020):
------------------------------------------------------------------------------------------------------------------------------------------
Thread 'oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020)' is waiting to acquire lock 'oracle.dfw.sampling.DumpSampling@2532a949' that is held by thread 'DFW Diagnostic Dump Sampling - DMSMetrics'
Stack trace:
------------
oracle.dfw.sampling.DumpSampling.getMetrics(DumpSampling.java:880)
oracle.dfw.sampling.DumpSamplingMetrics.refresh(DumpSamplingMetrics.java:202)
oracle.dms.spy.Metric.refresh(Metric.java:215)
oracle.dms.spy.Metric._getValues(Metric.java:293)
oracle.dms.spy.Metric.getSerializableValues(Metric.java:320)
oracle.dms.reporter.DMSTreeletGenerator.queryMetricValues(DMSTreeletGenerator.java:471)
oracle.dms.reporter.TreeletReporter.printGetResponse(TreeletReporter.java:243)
oracle.dms.reporter.TreeletReporter.printResponse(TreeletReporter.java:123)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2266)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2338)
oracle.dfw.impl.dump.DMSDump.executeDump(DMSDump.java:299)
oracle.dfw.impl.dump.DumpManagerImpl.executeDump(DumpManagerImpl.java:648)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$2.run(DiagnosticsDataExtractorImpl.java:2096)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$2.run(DiagnosticsDataExtractorImpl.java:2092)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
oracle.security.jps.internal.jaas.AccActionExecutor.execute(AccActionExecutor.java:70)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:282)
oracle.security.jps.internal.jaas.CascadeActionExecutor$SubjectPrivilegedExceptionAction.run(CascadeActionExecutor.java:82)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:370)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
weblogic.security.Security.runAs(Security.java:62)
oracle.security.jps.wls.jaas.WlsActionExecutor.execute(WlsActionExecutor.java:50)
oracle.security.jps.internal.jaas.CascadeActionExecutor.execute(CascadeActionExecutor.java:55)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:282)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity.executeAs(AbstractSubjectSecurity.java:236)
oracle.dfw.spi.weblogic.SecurityContextImpl.executeAs(SecurityContextImpl.java:129)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$3.run(DiagnosticsDataExtractorImpl.java:2109)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$3.run(DiagnosticsDataExtractorImpl.java:2105)
java.security.AccessController.doPrivileged(Native Method)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor.call(DiagnosticsDataExtractorImpl.java:2103)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor.call(DiagnosticsDataExtractorImpl.java:2008)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
[deadlocked thread] DFW Diagnostic Dump Sampling - DMSMetrics:
-------------------------------------------------------------
Thread 'DFW Diagnostic Dump Sampling - DMSMetrics' is waiting to acquire lock 'oracle.dfw.sampling.DumpSamplingMetrics@1b6796aa' that is held by thread 'oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020)'
Stack trace:
------------
oracle.dfw.sampling.DumpSamplingMetrics.refresh(DumpSamplingMetrics.java:197)
oracle.dms.spy.Metric.refresh(Metric.java:215)
oracle.dms.spy.Metric._getValues(Metric.java:293)
oracle.dms.spy.Metric.getSerializableValues(Metric.java:320)
oracle.dms.reporter.DMSTreeletGenerator.queryMetricValues(DMSTreeletGenerator.java:471)
oracle.dms.reporter.TreeletReporter.printGetResponse(TreeletReporter.java:243)
oracle.dms.reporter.TreeletReporter.printResponse(TreeletReporter.java:123)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2266)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2338)
oracle.dfw.impl.dump.DMSDump.executeDump(DMSDump.java:299)
oracle.dfw.sampling.DumpSampling.executeDump(DumpSampling.java:615)
oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:565)
.>
<Sep 15, 2020 3:23:52,968 AM EDT> <Critical> <Health> <BEA-310006> <Critical subsystem core in partition DOMAIN has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>
<Sep 15, 2020 3:23:52,969 AM EDT> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: [Health:310006]Critical subsystem core in partition DOMAIN has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>
<Sep 15, 2020 3:23:53,128 AM EDT> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at 'Tue Sep 15 03:23:53 2020'
[deadlocked thread] oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020):
------------------------------------------------------------------------------------------------------------------------------------------
Thread 'oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020)' is waiting to acquire lock 'oracle.dfw.sampling.DumpSampling@2532a949' that is held by thread 'DFW Diagnostic Dump Sampling - DMSMetrics'
Stack trace:
------------
oracle.dfw.sampling.DumpSampling.getMetrics(DumpSampling.java:880)
oracle.dfw.sampling.DumpSamplingMetrics.refresh(DumpSamplingMetrics.java:202)
oracle.dms.spy.Metric.refresh(Metric.java:215)
oracle.dms.spy.Metric._getValues(Metric.java:293)
oracle.dms.spy.Metric.getSerializableValues(Metric.java:320)
oracle.dms.reporter.DMSTreeletGenerator.queryMetricValues(DMSTreeletGenerator.java:471)
oracle.dms.reporter.TreeletReporter.printGetResponse(TreeletReporter.java:243)
oracle.dms.reporter.TreeletReporter.printResponse(TreeletReporter.java:123)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2266)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2338)
oracle.dfw.impl.dump.DMSDump.executeDump(DMSDump.java:299)
oracle.dfw.impl.dump.DumpManagerImpl.executeDump(DumpManagerImpl.java:648)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$2.run(DiagnosticsDataExtractorImpl.java:2096)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$2.run(DiagnosticsDataExtractorImpl.java:2092)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
oracle.security.jps.internal.jaas.AccActionExecutor.execute(AccActionExecutor.java:70)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:282)
oracle.security.jps.internal.jaas.CascadeActionExecutor$SubjectPrivilegedExceptionAction.run(CascadeActionExecutor.java:82)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:370)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
weblogic.security.Security.runAs(Security.java:62)
oracle.security.jps.wls.jaas.WlsActionExecutor.execute(WlsActionExecutor.java:50)
oracle.security.jps.internal.jaas.CascadeActionExecutor.execute(CascadeActionExecutor.java:55)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:282)
oracle.security.jps.internal.jaas.AbstractSubjectSecurity.executeAs(AbstractSubjectSecurity.java:236)
oracle.dfw.spi.weblogic.SecurityContextImpl.executeAs(SecurityContextImpl.java:129)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$3.run(DiagnosticsDataExtractorImpl.java:2109)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor$3.run(DiagnosticsDataExtractorImpl.java:2105)
java.security.AccessController.doPrivileged(Native Method)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor.call(DiagnosticsDataExtractorImpl.java:2103)
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$SingleDumpExecutor.call(DiagnosticsDataExtractorImpl.java:2008)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
[deadlocked thread] DFW Diagnostic Dump Sampling - DMSMetrics:
-------------------------------------------------------------
Thread 'DFW Diagnostic Dump Sampling - DMSMetrics' is waiting to acquire lock 'oracle.dfw.sampling.DumpSamplingMetrics@1b6796aa' that is held by thread 'oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Tue Sep 15 03:23:14 EDT 2020)'
Stack trace:
------------
oracle.dfw.sampling.DumpSamplingMetrics.refresh(DumpSamplingMetrics.java:197)
oracle.dms.spy.Metric.refresh(Metric.java:215)
oracle.dms.spy.Metric._getValues(Metric.java:293)
oracle.dms.spy.Metric.getSerializableValues(Metric.java:320)
oracle.dms.reporter.DMSTreeletGenerator.queryMetricValues(DMSTreeletGenerator.java:471)
oracle.dms.reporter.TreeletReporter.printGetResponse(TreeletReporter.java:243)
oracle.dms.reporter.TreeletReporter.printResponse(TreeletReporter.java:123)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2266)
oracle.dms.instrument.Noun.dumpMetrics(Noun.java:2338)
oracle.dfw.impl.dump.DMSDump.executeDump(DMSDump.java:299)
oracle.dfw.sampling.DumpSampling.executeDump(DumpSampling.java:615)
oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:565)
.>
<Sep 15, 2020 3:23:52,968 AM EDT> <Critical> <Health> <BEA-310006> <Critical subsystem core in partition DOMAIN has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>
<Sep 15, 2020 3:23:52,969 AM EDT> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: [Health:310006]Critical subsystem core in partition DOMAIN has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>
<Sep 15, 2020 3:23:53,128 AM EDT> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at 'Tue Sep 15 03:23:53 2020'
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |