Messages Are Getting Stuck and Not Reaching The SARM (Doc ID 1495028.1)

Last updated on JANUARY 19, 2017

Applies to:

Oracle Communications ASAP - Version 5.2.3 and later
Information in this document applies to any platform.
***Checked for Currency 19-08-2015***

Symptoms

On ASAP 5.2.3 the following was observerd,

Messages are getting Stuck at the weblogic SRT queue and frequently the user faces the problem of messages not reaching the SARM. Around 6k orders are pending at SRT queue in this example.

From the weblogic Logs:

 

Error Message:

 

####<Aug 22, 2012 11:18:12 AM IST> <Error> <WebLogicServer> <ggnpro21> <AdminServer> <[ACTIVE] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1345614492005> <BEA-000337> <[STUCK] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,855" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@fb103e", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:

                java.lang.Object.wait(Native Method)

                java.lang.Object.wait(Object.java:474)

                weblogic.common.CompletionRequest.getResult(CompletionRequest.java:109)

                weblogic.store.gxa.internal.GXATransactionImpl.commitStoreIO(GXATransactionImpl.java:99)

                weblogic.store.gxa.internal.GXATransactionImpl.doOperationCallbacks(GXATransactionImpl.java:215)

                weblogic.store.gxa.internal.GXAResourceImpl.commit(GXAResourceImpl.java:1448)

                weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1333)

                ….

….

….

…..>

####<Aug 22, 2012 11:18:12 AM IST> <Error> <WebLogicServer> <ggnpro21> <AdminServer> <[ACTIVE] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1345614492037> <BEA-000337> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,855" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@c69a5a", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:

                java.lang.Object.wait(Native Method)

                java.lang.Object.wait(Object.java:474)

                weblogic.messaging.dispatcher.Request.sleepTillNotified(Request.java:324)

                weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:801)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:129)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:112)

                weblogic.messaging.dispatcher.Request.dispatchAsync(Request.java:1046)

                weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:72)

                weblogic.jms.frontend.FEProducer.doDispatch(FEProducer.java:865)

                weblogic.jms.frontend.FEProducer.sendSingleDestination(FEProducer.java:1146)

                weblogic.jms.frontend.FEProducer.send(FEProducer.java:1360)

                weblogic.jms.frontend.FEProducer.invoke(FEProducer.java:1419)

                weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:759)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchSyncNoTran(DispatcherImpl.java:246)

                weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)

                weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1061)

                weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)

                weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)

                weblogic.jms.client.JMSProducer.send(JMSProducer.java:308)

                weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1025)

                weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:94)

                com.mslv.asap.activation.srp.MessageSender.sendTextReply(MessageSender.java:176)

                com.mslv.asap.oss.activation.XMLRequestBean.publishMessage(XMLRequestBean.java:1977)

                com.mslv.asap.oss.activation.XMLRequestBean.doCreateOrderByValue(XMLRequestBean.java:503)

                com.mslv.asap.oss.activation.XMLRequestBean.onMessage(XMLRequestBean.java:236)

                weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)

                weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)

                weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)

                weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)

                weblogic.jms.client.JMSSession.execute(JMSSession.java:3953)

                weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)

                weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)

                weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

                weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

>

####<Aug 22, 2012 11:18:12 AM IST> <Error> <WebLogicServer> <ggnpro21> <AdminServer> <[ACTIVE] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1345614492049> <BEA-000337> <[STUCK] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,855" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@bfcbbc", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:

                java.lang.Throwable.fillInStackTrace(Native Method)

                java.lang.Throwable.<init>(Throwable.java:181)

                java.lang.Exception.<init>(Exception.java:29)

                java.lang.RuntimeException.<init>(RuntimeException.java:32)

                java.util.NoSuchElementException.<init>(NoSuchElementException.java:28)

                java.util.StringTokenizer.nextToken(StringTokenizer.java:332)

                java.util.StringTokenizer.nextToken(StringTokenizer.java:360)

                com.sybase.jdbc2.jdbc.SybStatement.handleCallBody(SybStatement.java:2041)

                com.sybase.jdbc2.jdbc.SybStatement.processEscapes(SybStatement.java:351)

                com.sybase.jdbc2.jdbc.SybStatement.doEscapeProcessing(SybStatement.java:405)

                com.sybase.jdbc2.jdbc.SybPreparedStatement.<init>(SybPreparedStatement.java:56)

                com.sybase.jdbc2.jdbc.SybCallableStatement.<init>(SybCallableStatement.java:44)

                com.sybase.jdbc2.jdbc.SybConnection.prepareCall(SybConnection.java:954)

                com.sybase.jdbc2.jdbc.SybConnectionProxy.prepareCall(SybConnectionProxy.java:456)

                weblogic.jdbc.wrapper.PooledConnection.prepareCall(PooledConnection.java:293)

                weblogic.jdbc.common.internal.ConnectionEnv.makeStatement(ConnectionEnv.java:1247)

                weblogic.jdbc.common.internal.ConnectionEnv.getCachedStatement(ConnectionEnv.java:970)

                weblogic.jdbc.common.internal.ConnectionEnv.getCachedStatement(ConnectionEnv.java:958)

                weblogic.jdbc.wrapper.Connection.prepareCall(Connection.java:546)

                com.mslv.asap.activation.srp.dao.jdbc.JdbcWorkOrderHandlerDAO.beginWorkOrder(JdbcWorkOrderHandlerDAO.java:77)

                com.mslv.asap.activation.srp.SarmProxy._submitOrder(SarmProxy.java:727)

                com.mslv.asap.activation.srp.SarmProxy.submitOrder(SarmProxy.java:675)

                com.mslv.asap.oss.activation.JVTActivationBean.createOrderByValue(JVTActivationBean.java:579)

                com.mslv.asap.oss.activation.JVTActivation_ccenww_EOImpl.createOrderByValue(JVTActivation_ccenww_EOImpl.java:1040)

                com.mslv.asap.oss.activation.XMLRequestBean.doCreateOrderByValue(XMLRequestBean.java:494)

                com.mslv.asap.oss.activation.XMLRequestBean.onMessage(XMLRequestBean.java:236)

                weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)

                weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)

                weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)

                weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)

                weblogic.jms.client.JMSSession.execute(JMSSession.java:3953)

                weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)

                weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)

                weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

                weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

 

>

####<Aug 22, 2012 11:18:12 AM IST> <Error> <WebLogicServer> <ggnpro21> <AdminServer> <[ACTIVE] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1345614492064> <BEA-000337> <[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,855" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@1d8b470", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:

                java.lang.Object.wait(Native Method)

                java.lang.Object.wait(Object.java:474)

                weblogic.messaging.dispatcher.Request.sleepTillNotified(Request.java:324)

                weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:801)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:129)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:112)

                weblogic.messaging.dispatcher.Request.dispatchAsync(Request.java:1046)

                weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:72)

                weblogic.jms.frontend.FEProducer.doDispatch(FEProducer.java:865)

                weblogic.jms.frontend.FEProducer.sendSingleDestination(FEProducer.java:1146)

                weblogic.jms.frontend.FEProducer.send(FEProducer.java:1360)

                weblogic.jms.frontend.FEProducer.invoke(FEProducer.java:1419)

                weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:759)

                weblogic.messaging.dispatcher.DispatcherImpl.dispatchSyncNoTran(DispatcherImpl.java:246)

                weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:59)

                weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1061)

                weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:670)

                weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:455)

                weblogic.jms.client.JMSProducer.send(JMSProducer.java:308)

                weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1025)

                weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:94)

                com.mslv.asap.activation.srp.MessageSender.sendTextReply(MessageSender.java:176)

                com.mslv.asap.oss.activation.XMLRequestBean.publishMessage(XMLRequestBean.java:1977)

                com.mslv.asap.oss.activation.XMLRequestBean.doCreateOrderByValue(XMLRequestBean.java:503)

                com.mslv.asap.oss.activation.XMLRequestBean.onMessage(XMLRequestBean.java:236)

                weblogic.ejb.container.internal.MDListener.execute(MDListener.java:429)

                weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)

                weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)

                weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4060)

                weblogic.jms.client.JMSSession.execute(JMSSession.java:3953)

                weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4467)

                weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)

                weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

                weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

>

####<Aug 22, 2012 12:15:41 PM IST> <Error> <JDBC> <ggnpro21> <AdminServer> <[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1345617941115> <BEA-001112> <Test "ping" set up for pool "System.prod.ApplicationType.ServiceActivation.Application.1-0;5-2;ASAP.Comp.RPCConnectionPool" failed with exception: "java.sql.SQLException: JZ006: Caught IOException: com.sybase.jdbc2.jdbc.SybConnectionDeadException: JZ0C0: Connection is already closed.".>

 

 

Changes

 None                                                                          

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms