ESB Perfomrance Issue Due To Very High Number Of HTTPClient.StreamDemultiplexor Locked Threads (Doc ID 1371060.1)

Last updated on JUNE 01, 2017

Applies to:

Oracle ESB - Version: 10.1.3.4 and later   [Release: AS10gR3 and later ]
Information in this document applies to any platform.

Symptoms

On ESB 10.1.3.4
In a cluster environment, a degration of the performance on ESB/SOA is causing the system to freeze, JVM is crashing frequently, there is not OOM or relevant errors on the oc4j_soa log files, The only error that occurred multiple times is the following one:

Process Ping Failed: HTTP_Server~HTTP_Server~HTTP_Server~1 (1635261662:8173) [The connection receive timed out]

Thread dumps shown the following thread hundred of times:


"AJPRequestHandler-HTTPThreadGroup-423" prio=10 tid=0x020f5210 nid=0x128b runnable [0x4867d000..0x4867fbf0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:204)
at HTTPClient.BufferedInputStream.read(BufferedInputStream.java:124)
at HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:338)
- locked <0x87b46a20> (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:163)
- locked <0x87b46a20> (a HTTPClient.StreamDemultiplexor)
at HTTPClient.RespInputStream.read(RespInputStream.java:125)
at HTTPClient.Response.readResponseHeaders(Response.java:1026)
at HTTPClient.Response.getHeaders(Response.java:740)
- locked <0x87b46a20> (a HTTPClient.StreamDemultiplexor)
at HTTPClient.Response.getStatusCode(Response.java:276)
at HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:92)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:773)
- locked <0x87b46b50> (a HTTPClient.HTTPResponse)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:206)
at HTTPClient.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.post(HttpSOAPConnection.java:417)
at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection$PriviledgedPost.run(HttpSOAPConnection.java:953)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.call(HttpSOAPConnection.java:154)
at oracle.j2ee.ws.client.http.HttpClientTransport.invokeImpl(HttpClientTransport.java:167)
at oracle.j2ee.ws.client.http.HttpClientTransport.invoke(HttpClientTransport.java:150)
at oracle.j2ee.ws.client.StreamingSender._sendImpl(StreamingSender.java:176)
at oracle.j2ee.ws.client.StreamingSender._send(StreamingSender.java:113)
at oracle.j2ee.ws.client.dii.CallInvokerImpl.directInvoke(CallInvokerImpl.java:760)
at oracle.j2ee.ws.client.dii.BasicCall.directInvoke(BasicCall.java:754)
at oracle.j2ee.ws.client.dii.BasicCall.invoke(BasicCall.java:660)
at com.collaxa.cube.ws.wsif.providers.oc4j.jaxrpc.WSIFOperation_JaxRpc.invokeOperation(WSIFOperation_JaxRpc.java:1850)
at com.collaxa.cube.ws.wsif.providers.oc4j.jaxrpc.WSIFOperation_JaxRpc.invokeRequestResponseOperation(WSIFOperation_JaxRpc.java:1570)
at com.collaxa.cube.ws.wsif.providers.oc4j.jaxrpc.WSIFOperation_JaxRpc.executeRequestResponseOperation(WSIFOperation_JaxRpc.java:1233)
at oracle.tip.esb.server.common.wsif.WSIFInvoker.executeOperation(WSIFInvoker.java:923)
at oracle.tip.esb.server.common.wsif.WSIFInvoker.nextService(WSIFInvoker.java:825)
at oracle.tip.esb.server.common.wsif.WSIFInvoker.nextService(WSIFInvoker.java:848)
at oracle.tip.esb.server.service.impl.outadapter.OutboundAdapterService.nextService(OutboundAdapterService.java:231)
at oracle.tip.esb.server.service.impl.outadapter.OutboundAdapterService.processBusinessEvent(OutboundAdapterService.java:138)
at oracle.tip.esb.server.dispatch.InitialEventDispatcher.dispatchNonRoutingService(InitialEventDispatcher.java:407)
at oracle.tip.esb.server.dispatch.InitialEventDispatcher.dispatch(InitialEventDispatcher.java:165)
at oracle.tip.esb.server.dispatch.BusinessEvent.raise(BusinessEvent.java:1983)
at oracle.tip.esb.server.dispatch.BusinessEvent.raise(BusinessEvent.java:1470)
at oracle.tip.esb.utils.EventUtils.raiseBusinessEvent(EventUtils.java:215)
at oracle.tip.esb.utils.EventUtils.raiseBusinessEvent(EventUtils.java:139)
at oracle.tip.esb.server.service.EsbRouterSubscription.onBusinessEvent(EsbRouterSubscription.java:313)
at oracle.tip.esb.server.dispatch.EventDispatcher.executeSubscription(EventDispatcher.java:138)
at oracle.tip.esb.server.dispatch.InitialEventDispatcher.processSubscription(InitialEventDispatcher.java:547)
at oracle.tip.esb.server.dispatch.InitialEventDispatcher.processSubscriptions(InitialEventDispatcher.java:529)
at oracle.tip.esb.server.dispatch.EventDispatcher.dispatchRoutingService(EventDispatcher.java:94)
at oracle.tip.esb.server.dispatch.InitialEventDispatcher.dispatch(InitialEventDispatcher.java:161)
at oracle.tip.esb.server.dispatch.BusinessEvent.raise(BusinessEvent.java:1983)
at oracle.tip.esb.server.dispatch.BusinessEvent.raise(BusinessEvent.java:1470)
at oracle.tip.esb.server.service.impl.soap.EventOracleSoapProvider.raiseEvent(EventOracleSoapProvider.java:462)
at oracle.tip.esb.server.service.impl.soap.EventOracleSoapProvider.processMessage(EventOracleSoapProvider.java:240)
at oracle.j2ee.ws.server.provider.ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:956)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.provider.ProviderProcessor.doRequestProcessing(ProviderProcessor.java:466)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.WebServiceProcessor.doService(WebServiceProcessor.java:96)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:713)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms