osb_server1 Stops Processing Requests and Shows "ThreadPool Has Stuck Threads" (Doc ID 1512718.1)

Last updated on NOVEMBER 07, 2016

Applies to:

Oracle Service Bus - Version 11.1.1.4.0 and later
Information in this document applies to any platform.

Symptoms

You notice the OSB Server has stopped processing requests and the WLS Console shows the AdminServer and osb_server1 in a Warning State with "Thread pool has stuck threads"

An incident has been created for the issue and the readme.txt for the incident has:

(Incidents are written to <server_home>/adr/diag/ofm/<domainname>/<servername>/incident)

Incident Id: 3
Incident Source: SYSTEM
Create Time: Mon Oct 22 09:56:24 EDT 2012
Problem Key: BEA-337 [WebLogicServer]
Error Message Id: BEA-337
Description
----------- Incident detected using watch rule "StuckThread":
Watch time: Oct 22, 2012 9:56:23 AM EDT
Watch ServerName: osb_server1
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
Watch DomainName: domainName
Watch Data:
DATE : Oct 22, 2012 9:56:23 AM EDT
SERVER : osb_server1
MESSAGE : [STUCK] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "612" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@5a7c16d", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: Thread-147 "[STUCK] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" { -- Waiting for notification on: java.lang.Object@5a7c0d2[fat lock] java.lang.Object.wait(Object.java:485) com.bea.wli.sb.pipeline.PipelineContextImpl$SynchronousListener.waitForResponse(PipelineContextImpl.java:1563) com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:525) stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1385) stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:159) com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343) stages.transform.runtime.IfThenElseRuntimeStep.processMessage(IfThenElseRuntimeStep.java:74) com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:343) com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:31) com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84) com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1022) com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:140) com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1022) com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:51) com.bea.wli.sb.pipeline.Node.processMessage(Node.java:65) com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1022) com.bea.wli.sb.pipeline.Router.processMessage(Router.java:196) com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:35) com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593) com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:350) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:142) com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55) com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:572) com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:252) com.bea.wli.sb.transports.jms.JmsInboundMDB.onMessage(JmsInboundMDB.java:86) sun.reflect.GeneratedMethodAccessor1504.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:575) com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148) com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86) com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148) com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102) com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148) com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151) $Proxy210.onMessage(Unknown Source) weblogic.ejb.container.internal.MDListener.execute(MDListener.java:456) weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:405) weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:268) weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4638) weblogic.jms.client.JMSSession.execute(JMSSession.java:4134) weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3820) weblogic.jms.client.JMSSession.access$000(JMSSession.java:115) weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5170) weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:168) } SUBSYSTEM : WebLogicServer USERID : SEVERITY : Error THREAD : [ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID : BEA-000337 MACHINE : CMWP053 TXID : CONTEXTID : 2156fbeb9c869391:-59dcb561:13a51805f0c:-8000-000000000008b3cd TIMESTAMP : 1350914183517 Diagnostic Dumps ---------------- Name: jvm.threads Dump Files: jvm_threads1_i3.dmp Name: dms.metrics Dump Files: dms_metrics2_i3.dmp Name: odl.quicktrace Dump Files: odl_quicktrace3_i3.dmp Name: odl.logs Dump Arguments: ecid=2156fbeb9c869391:-59dcb561:13a51805f0c:-8000-000000000008b3cd Dump Files: odl_logs4_i3.dmp Name: odl.logs Dump Arguments: timestamp=1350914184080 Dump Files: odl_logs5_i3.dmp Name: wls.image Dump Files: diagnostic_image_osb_server1_2012_10_22_09_56_41.zip


The associated diagnostic dump files for the incident show multiple errors indicated in the Execute Threads. The errors are all like:

[2012-10-22T09:21:35.350-04:00] [osb_server1] [NOTIFICATION] [] [oracle.toplink.default] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 2156fbeb9c869391:-59dcb561:13a51805f0c:-8000-000000000008a56c,0] [APP: usermessagingserver] TopLink, version: Oracle TopLink - 11g Release 1 (11.1.1.4.0) (Build 101210)
[2012-10-22T09:21:35.350-04:00] [osb_server1] [NOTIFICATION] [] [oracle.toplink.default] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 2156fbeb9c869391:-59dcb561:13a51805f0c:-8000-000000000008a56c,0] [APP: usermessagingserver] Server: WebLogic Server 10.3.4.0 Fri Dec 17 20:47:33 PST 2010 1384255
[2012-10-22T09:21:35.350-04:00] [osb_server1] [ERROR] [] [oracle.toplink.default] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 2156fbeb9c869391:-59dcb561:13a51805f0c:-8000-000000000008a56c,0] [APP: usermessagingserver] [[
Local Exception Stack:
Exception [TOPLINK-7060] (Oracle TopLink - 11g Release 1 (11.1.1.4.0) (Build 101210)):
oracle.toplink.exceptions.ValidationException
Exception Description: Cannot acquire data source [jdbc/OraSDPMDataSource].
Internal Exception: javax.naming.NameNotFoundException: Unable to resolve 'jdbc.OraSDPMDataSource'.
Resolved 'jdbc'; remaining name 'OraSDPMDataSource'

Changes

 It's suspected that a change to the default configuration for the usermessagingserver application has occurred.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms