Reports Engine Hangs Under Heavy Load - Status The report is opening to run .. Waiting (Doc ID 1492468.1)

Last updated on NOVEMBER 30, 2016

Applies to:

Oracle Reports Developer - Version 11.1.1.4.0 and later
Information in this document applies to any platform.

Symptoms

Running hundreds or thousands of report requests simultaneously cause the report engine to hang. Reports Server continue to receive report requests and Report webcommands like showjobs or getserverinfo work fine but report engines wont process any new request.

If engineResponseTimeOut setting is enabled then you will see the following error in rwserver_diagnostic.log file

REP-56109: Job XXXX is terminated due to a connection timeout..

After getting above message the report engine will get re-started by the Reports Server and new requests will be sent to the engine until the new hanging situation will araise again and the same situation will be repeated again and again.

Example of this situation reading Reports trace files

rwserver_diagnostic.log

[2012-04-24T13:25:18.191-04:00] ..... JobManager.runJobLocal Job 7165 got Engine rwEng-0
[2012-04-24T13:25:18.191-04:00] ..... JobManager:runJobInEngine Job 7165 calling setCommand on engine rwEng-0
[2012-04-24T13:25:18.195-04:00] ..... EngineManager:updateEngineState Engine rwEng-0 status is 3
.
[2012-04-24T13:40:18.916-04:00] ..... JobManager:updateJobStatus Job 7165 status is: The report terminated with error: [[
REP-56109: Job 7165 is terminated due to a connection timeout..

From above trace information JOBID 7165 is assigned to Engine-0 at 13:25:18

rwEng-0_diagnostic.log

[2012-04-24T13:25:18.188-04:00] ..... rwsjni:CRunReport Job ID = 7164 Exit - jsprun
[2012-04-24T13:25:18.188-04:00] ..... EngineImpl:run After calling CRunReport()
[2012-04-24T13:25:18.188-04:00] ..... EngineImpl:run CRunReport returns: 0
[2012-04-24T13:25:18.188-04:00] ..... EngineImpl:run Before calling EngineComm.updateEngineState(STATE_READY)
[2012-04-24T13:25:18.193-04:00] ..... EngineImpl:setCommandLine Running
[2012-04-24T13:25:18.193-04:00] ..... EngineImpl:setCommandLine Before calling EngineComm.updateEngineState(STATE_RUNNING)

 

From above report engine-0 trace file one can see that there is no information regarding JOBID 7165 at 13:25:18. This means that the engine 0 is now hanging.

The expected line in Engine-0 trace file that is missing would be the following one

"EngineImpl:run  After calling EngineComm.updateEngineState(STATE_READY)", but that line wont be recorded.

Using jstack java utility to capture current state of Report Engine hanging situation will show the following stack trace

"p: default-threadpool; w: Idle" daemon prio=7 tid=6000000000ddf800 nid=19 lwp_id=1282217 in Object.wait() [87ffffff5ea00000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <87ffffffe2662240> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at com.sun.corba.se.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:142)
- locked <87ffffffe2662240> (a java.lang.Object)
at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1082)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:253)
at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:381)
at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:355)
at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:129)
at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
at oracle.reports.server._EngineCommStub.updateEngineState(_EngineCommStub.java:103)
at oracle.reports.engine.EngineImpl.run(EngineImpl.java:562)
at oracle.reports.engine.EngineClassPOA._invoke(EngineClassPOA.java:104)
at com.sun.corba.se.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:637)
at com.sun.corba.se.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:189)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1682)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1540)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:922)
at com.sun.corba.se.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:181)
at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:694)
at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.dispatch(SocketOrChannelConnectionImpl.java:449)
at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1211)
at com.sun.corba.se.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:471)
at com.sun.corba.se.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:500)

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms