WTC: "weblogic.wtc.gwt.TuxedoConnection.tpacall()" Was Blocked By "java.net.SocketOutputStream.socketWrite0()" (Doc ID 2207439.1)

Last updated on NOVEMBER 28, 2016

Applies to:

Oracle WebLogic Server - Version 10.3.6 to 12.1.1.0
Information in this document applies to any platform.

Symptoms

On : WebLogic 10.3.6 version, WebLogic Tuxedo Connector sub-component, with Oracle Service Bus(also known as OSB)

The following STUCK Threads are occurring:

"[STUCK] ExecuteThread: '956' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002af268f2b800 nid=0x739d waiting for monitor entry [0x00002af2b61db000]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.wtc.jatmi.dsession._tpacall_internal(dsession.java:3088)
- waiting to lock <0x0000000763762038> (a java.io.DataOutputStream)
at weblogic.wtc.jatmi.dsession.tprplycall(dsession.java:4636)
at weblogic.wtc.gwt.TuxedoConnection.tpacall(TuxedoConnection.java:417)
at com.bea.wli.sb.transports.tuxedo.TuxedoOutboundMessageContext.send(TuxedoOutboundMessageContext.java:408)
at com.bea.wli.sb.transports.tuxedo.TuxedoTransportProvider.sendMessageAsync(TuxedoTransportProvider.java:470)
at sun.reflect.GeneratedMethodAccessor453.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
at com.sun.proxy.$Proxy126.sendMessageAsync(Unknown Source)
...  ...
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.securedInvoke(RequestHelperBase.java:159)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.service(RequestHelperBase.java:107)
at com.bea.wli.sb.transports.http.wls.HttpTransportServlet.service(HttpTransportServlet.java:129)
at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3732)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)



"[STUCK] ExecuteThread: '471' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00002af268d9e800 nid=0x64c4 runnable [0x00002af293d38000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at weblogic.wtc.jatmi.TpeOut.writeTuxedoMessage(TpeOut.java:240)
at weblogic.wtc.jatmi.TpeOut.write(TpeOut.java:386)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked <0x0000000763762038> (a java.io.DataOutputStream)
at weblogic.wtc.jatmi.tfmh.write_tfmh(tfmh.java:815)
at weblogic.wtc.jatmi.dsession._tpacall_internal(dsession.java:3093)
- locked <0x0000000763762038> (a java.io.DataOutputStream)
at weblogic.wtc.jatmi.dsession.tprplycall(dsession.java:4636)
at weblogic.wtc.gwt.TuxedoConnection.tpacall(TuxedoConnection.java:417)
at com.bea.wli.sb.transports.tuxedo.TuxedoOutboundMessageContext.send(TuxedoOutboundMessageContext.java:408)
at com.bea.wli.sb.transports.tuxedo.TuxedoTransportProvider.sendMessageAsync(TuxedoTransportProvider.java:470)
at sun.reflect.GeneratedMethodAccessor453.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

... ...
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.securedInvoke(RequestHelperBase.java:159)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.service(RequestHelperBase.java:107)
at com.bea.wli.sb.transports.http.wls.HttpTransportServlet.service(HttpTransportServlet.java:129)
at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3732)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

 

Came to the point that the WLI servers, for OSB, were not responding at all as a synchronized SocketWrite() operation never did wake up (might be because remote machine DOWN/CRASHED, TCP window full mechanism, Firewall misconfiguration, ...).

Resulted in the application server spawning more and more threads. All threads hanging on the synchronized SocketWrite operation.
 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms