SOA 11g: "STUCK" Threads due to Contention in the Classes HTTPClient/StreamDemultiplexor Result in Server Performance Issues
(Doc ID 1564631.1)
Last updated on FEBRUARY 04, 2019
Applies to:
Oracle SOA Suite - Version 11.1.1.2.0 to 11.1.1.9.0 [Release 11gR1 to 11g]Information in this document applies to any platform.
Symptoms
Reviewing the incident folder of BPEL Servers:
<DOMAIN_NAME>/servers/<SERVER_NAME>/adr/diag/ofm/<DOMAIN_NAME>/<SERVER_ANAME>/incident/incdir_XX
It can be found some Threads in which the following contention (locks) is generated:
The Blocking Thread is reading from an HTTP connections using StreamDemultiplexor class (partial StackTrace):
"[STUCK] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" id=284 idx=0x23c tid=24922 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at HTTPClient/BufferedInputStream.fillBuff(BufferedInputStream.java:206)
at HTTPClient/BufferedInputStream.read(BufferedInputStream.java:126)[optimized]
at HTTPClient/StreamDemultiplexor.read(StreamDemultiplexor.java:356)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[recursive]
at HTTPClient/RespInputStream.read(RespInputStream.java:151)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[recursive]
at HTTPClient/RespInputStream.read(RespInputStream.java:112)
at HTTPClient/Response.readResponseHeaders(Response.java:1144)
at HTTPClient/Response.getHeaders(Response.java:867)[inlined]
at HTTPClient/Response.getStatusCode(Response.java:331)[inlined]
at HTTPClient/RetryModule.responsePhase1Handler(RetryModule.java:94)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[thin lock]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[thin lock]
at HTTPClient/HTTPResponse.handleResponseImpl(HTTPResponse.java:872)[optimized]
at HTTPClient/HTTPResponse.access$000(HTTPResponse.java:62)[optimized]
at HTTPClient/HTTPResponse$2.run(HTTPResponse.java:839)
at HTTPClient/HTTPResponse$2.run(HTTPResponse.java:837)
at HTTPClient/HttpClientConfiguration.doAction(HttpClientConfiguration.java:698)[optimized]
at HTTPClient/HTTPResponse.handleResponse(HTTPResponse.java:837)
^-- Holding lock: HTTPClient/HTTPResponse@0x1e1748250[biased lock]
at HTTPClient/HTTPResponse.getStatusCode(HTTPResponse.java:242)
at HTTPClient/HttpURLConnection.getResponseCode(HttpURLConnection.java:330)
…
…
…
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1491)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:256)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at HTTPClient/BufferedInputStream.fillBuff(BufferedInputStream.java:206)
at HTTPClient/BufferedInputStream.read(BufferedInputStream.java:126)[optimized]
at HTTPClient/StreamDemultiplexor.read(StreamDemultiplexor.java:356)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[recursive]
at HTTPClient/RespInputStream.read(RespInputStream.java:151)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[recursive]
at HTTPClient/RespInputStream.read(RespInputStream.java:112)
at HTTPClient/Response.readResponseHeaders(Response.java:1144)
at HTTPClient/Response.getHeaders(Response.java:867)[inlined]
at HTTPClient/Response.getStatusCode(Response.java:331)[inlined]
at HTTPClient/RetryModule.responsePhase1Handler(RetryModule.java:94)[optimized]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[thin lock]
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1b2e21900[thin lock]
at HTTPClient/HTTPResponse.handleResponseImpl(HTTPResponse.java:872)[optimized]
at HTTPClient/HTTPResponse.access$000(HTTPResponse.java:62)[optimized]
at HTTPClient/HTTPResponse$2.run(HTTPResponse.java:839)
at HTTPClient/HTTPResponse$2.run(HTTPResponse.java:837)
at HTTPClient/HttpClientConfiguration.doAction(HttpClientConfiguration.java:698)[optimized]
at HTTPClient/HTTPResponse.handleResponse(HTTPResponse.java:837)
^-- Holding lock: HTTPClient/HTTPResponse@0x1e1748250[biased lock]
at HTTPClient/HTTPResponse.getStatusCode(HTTPResponse.java:242)
at HTTPClient/HttpURLConnection.getResponseCode(HttpURLConnection.java:330)
…
…
…
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1491)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:256)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
The Blocked Threads are trying to close the HTTPClient.ResInputStream objects, reusing the sockets and resulting in the consumption of many threads, delaying processing of requests:
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x60 tid=306 prio=1 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: HTTPClient/StreamDemultiplexor@0x1b2c1e288[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Locks.java:955)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1083)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at HTTPClient/RespInputStream.close(RespInputStream.java:241)
at HTTPClient/RespInputStream$RISClosingTask.run(RespInputStream.java:414)
at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:256)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
-- Blocked trying to get lock: HTTPClient/StreamDemultiplexor@0x1b2c1e288[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Locks.java:955)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1083)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at HTTPClient/RespInputStream.close(RespInputStream.java:241)
at HTTPClient/RespInputStream$RISClosingTask.run(RespInputStream.java:414)
at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:256)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:221)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |