Weblogic WLS Destination Unreachable When Trying To Enqueue Messages To Tuxedo. (Doc ID 2211765.1)

Last updated on DECEMBER 07, 2016

Applies to:

Oracle WebLogic Server - Version 12.1.3.0.0 and later
Information in this document applies to any platform.

Symptoms

On : WebLogic 12.1.3.0.0 version
WebLogic Server(also known as WLS) is not responding to connections to the AdminServer. Managed Server logs indicate destination unreachable.

Application uses WebLogic Tuxedo Connector(also known as WTC)  to interface between parts of the application in a Tuxedo domain and parts in the WebLogic domain.   Both domains run on the same server.

Looking in logs for all ManagedServers in the time frame leading up to ManagedServer2 and ManagedServer1 failing, it is seen that most of the stuck threads are the result of the following exception recorded when enqueuing a message in the Tuxedo domain:

####<Nov 9, 2016 3:35:02 PM UTC> <Error> <Default> <wlsmachine> <ManagedServer0> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <TUXSIDE> <BEA1-57EEA7229DA859ADC149> <fec78e22-63e3-45cb-a8f6-0cc553fe8cab-0001b3e3> <1478705702877> <BEA-000000> <Exception caught for [ResourceBean_eegsn8_Impl.signOn], Transaction=[BEA1-57EEA7229DA859ADC149
], Correlation=[null], Principals=[TUXSIDE, Mobiles]

...
java.lang.IllegalStateException: Failed to enqueue msg=weblogic.wtc.jatmi.TypedCArray@cf3b2933
at com.myco.servicesuite.util.Debug.wrap(Debug.java:115)
at com.myco.servicesuite.util.tuxqueue.TuxQueueSender.enqueue(TuxQueueSender.java:212)
at com.myco.servicesuite.util.tuxqueue.TuxQueueSender.post(TuxQueueSender.java:167)
at com.myco.servicesuite.util.tuxqueue.NotQueueSender.send(NotQueueSender.java:41)
...

####<Nov 9, 2016 3:37:59 PM UTC> <Error> <WebLogicServer> <wlsmachine> <ManagedServer0> <[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <fec78e22-63e3-45cb-a8f6-0cc553fe8cab-0001b48a> <1478705879262> <BEA-000337> <[STUCK] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "172" seconds working on the request "Workmanager: default, Version: 0, Scheduled=false, Started=true, Started time: 172982 ms
", which is more than the configured time (StuckThreadMaxTime) of "120" seconds in "server-failure-trigger". Stack trace:
com.myco.servicesuite.util.tuxqueue.TuxQueueSender.enqueue(TuxQueueSender.java:195)
com.myco.servicesuite.util.tuxqueue.TuxQueueSender.post(TuxQueueSender.java:167)
com.myco.servicesuite.util.tuxqueue.NotQueueSender.send(NotQueueSender.java:41)
com.myco.servicesuite.util.MylogBuilder.post(MylogBuilder.java:236)
com.myco.servicesuite.util.MylogBuilder.post(MylogBuilder.java:220)
...


In the same time frame, in the Tuxedo domain application logs we see jatmi call failures when making java calls from the Tuxedo domain to the WebLogic domain:

09-07:36:16.689 (InfTuxBase.cpp@[InfTuxBase] 1098) Request to service [JatmiSvc] failed. tperrno [13], msg [TPETIME - timeout occured], tperrordetail [No details], flags [0]
09-07:36:16.689 (JsiOrderProgressionInterface.cpp@[global] 319) Jsi cannot call service handler: OrderProgression.processCompletionData()
09-07:36:16.689 (InfDecoder.h@[InfDecoder] 450) JsiReturn cannot get return for [JsiResult]
09-07:36:16.690 (MrepMmQueueHandler_obj.cpp@[MrepMmQueueHandler_obj] 358) DbugExceptionException=TPED_INVALID_XA_TRANSACTION
09-07:36:16.690 (MrepMmQueueHandler_atmi.h@[global] 108) Caught DbugException [TPED_INVALID_XA_TRANSACTION]

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms