OSB Applications Hanging Due To Huge Load Average In Server (Doc ID 2206025.1)

Last updated on NOVEMBER 21, 2016

Applies to:

Oracle Service Bus - Version 11.1.1.7.0 and later
Information in this document applies to any platform.

Symptoms

The issue is whenever user is trying to start any OSB WebLogic domain server (does not matter if it is Admin server or any OSB managed server), the start sequence is taking unusually long time - 50-60 minutes as compared to the usual 3-4 minutes. Also, during the start sequence the load average of the physical node is bumping up rapidly leading to slow performance of the entire physical node.

User does not see the server going to Hung or Crashing - the only exception being only when the managed server is getting up, the admin server once goes to WARNING state due to STUCK threads since the managed server takes too much time to start.

In the AdminServer.out seeing the following error:

<Nov 8, 2016 5:32:05 AM EST> <Notice> <Stdout> <BEA-000000> <<Nov 8, 2016 5:32:05 AM EST> <Emergency> <oracle.dfw.incident> <BEA-000000> <incident 2 created with problem key "BEA-000337 [WebLogicServer]">>
<Nov 8, 2016 5:33:01 AM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "691" seconds working on the request "Workmanager: weblogic.kernel.Default, Version: 0, Scheduled=true, Started=true, Started time: 691798 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:152)
java.net.SocketInputStream.read(SocketInputStream.java:122)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:159)
java.io.InputStream.read(InputStream.java:101)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
java.io.InputStreamReader.read(InputStreamReader.java:184)
java.io.BufferedReader.fill(BufferedReader.java:154)
java.io.BufferedReader.readLine(BufferedReader.java:317)
java.io.BufferedReader.readLine(BufferedReader.java:382)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:289)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:314)
weblogic.nodemanager.client.NMServerClient.start(NMServerClient.java:102)
weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:47)
weblogic.kernel.WorkManagerWrapper$1.run(WorkManagerWrapper.java:63)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms