OC4J Container Appears to be Hung and Restarted by OPMN Process (Doc ID 1413711.1)

Last updated on DECEMBER 05, 2016

Applies to:

Oracle Containers for J2EE - Version 10.1.3.4.0 and later
Information in this document applies to any platform.
***Checked for relevance on 16-Nov-2016***

Symptoms

In Oracle AS 10.1.3.4 some of the OC4J container instances appears to be getting hung, and after a while it is being restarted by the OPMN non-responsive process monitoring protocol.
In this specific case it was OC4J_WSRP the one getting hung , but it could be any other.
The configuration is an OPMN based cluster (node1 and node2) where the OC4J component is replicating HTTP_Session status. 

After a restart of the involved OC4J instances, they go to healthy state again and keep serving requests. 

The analysis of a sequence of Thread dumps of the JVM show the following information 

Threads from Node1

749 threads of which 74% are stuck in monitor wait, which means they are waiting on "other threads" to finish. The contention observed is:

1) Big contention in object <0x7173d1d0> (a com.evermind.server.http.HttpSite)

Locking thread

"TaskManager" prio=10 tid=0x003f7750 nid=0xbf in Object.wait() [0x3f77f000..0x3f77f9f0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:199)
- locked <0x718e95f0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.InterruptableTestAndWaitLock.doAcquire(InterruptableTestAndWaitLock.java:63)
- locked <0x718e95f0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.TestAndWaitLock.acquire(TestAndWaitLock.java:126)
at com.oracle.bricks.Manager.reapReplicas(Manager.java:692)
at com.evermind.server.http.JGroupHttpService.checkSessionTimeouts(JGroupHttpService.java:122)
at com.evermind.server.http.HttpApplication.checkSessionTimeouts(HttpApplication.java:6781)
at com.evermind.server.http.HttpSite.checkSessionTimeouts(HttpSite.java:937)
- locked <0x7173d1d0> (a com.evermind.server.http.HttpSite)
at com.evermind.server.http.SessionTimeoutTask.checkSessionTimeouts(SessionTimeoutTask.java:156)
at com.evermind.server.http.SessionTimeoutTask.run(SessionTimeoutTask.java:126)
at com.evermind.util.Task.schedule(Task.java:51)
at com.evermind.util.TaskManager.run(TaskManager.java:212)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:298)
at java.lang.Thread.run(Thread.java:595)

392 threads waiting on acquiring object <0x7173d1d0> with pattern

"AJPRequestHandler-HTTPThreadGroup-246" prio=10 tid=0x01059330 nid=0x278 waiting for monitor entry [0x35cfe000..0x35cff970]
at com.evermind.server.http.HttpSite.getRequestDispatcher(HttpSite.java:343)
- waiting to lock <0x7173d1d0> (a com.evermind.server.http.HttpSite)
at com.evermind.server.http.HttpApplication.unprivileged_getRequestDispatcher(HttpApplication.java:1687)

at com.evermind.server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:1672)
at oracle.portlet.server.containerimpl.PortletApplicationImpl.getRequestDispatcher(PortletApplicationImpl.java:713)
at portal.web.portlets.NewsPortlet.doView(NewsPortlet.java:58)
at javax.portlet.GenericPortlet.doDispatch(GenericPortlet.java:235)
at javax.portlet.GenericPortlet.render(GenericPortlet.java:163)
at oracle.portlet.server.containerimpl.ServerImpl.getMarkup(ServerImpl.java:166)
at oracle.portlet.wsrp.v1.WSRPv1ToServer.getMarkup(WSRPv1ToServer.java:5581)
at oracle.portlet.wsrp.v1.WSRP_v1_Markup_PortTypeSoapToJaxb.getMarkup(WSRP_v1_Markup_PortTypeSoapToJaxb.java:68)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.invoke_getMarkup(WSRP_v1_Markup_Binding_SOAP_Tie.java:157)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.processingHook(WSRP_v1_Markup_Binding_SOAP_Tie.java:782)
at oracle.j2ee.ws.server.StreamingHandler.handle(StreamingHandler.java:297)
at oracle.j2ee.ws.server.JAXRPCProcessor.doEndpointProcessing(JAXRPCProcessor.java:413)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.JAXRPCProcessor.doRequestProcessing(JAXRPCProcessor.java:277)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.JAXRPCProcessor.doService(JAXRPCProcessor.java:134)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
...

2) Big contention as well in object  <0x7173d1a8> (a java.util.HashMap)

Locking thread is

"AJPRequestHandler-HTTPThreadGroup-159" prio=10 tid=0x00fc0bb8 nid=0x1f1 in Object.wait() [0x43ffe000..0x43fff8f0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:199)
- locked <0x718e95f0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.InterruptableTestAndWaitLock.doAcquire(InterruptableTestAndWaitLock.java:63)
- locked <0x718e95f0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.TestAndWaitLock.acquire(TestAndWaitLock.java:126)
at com.oracle.bricks.Manager.add(Manager.java:345)
at com.oracle.bricks.j2ee.SessionManager.create(SessionManager.java:77)
at com.evermind.server.http.JGroupHttpService.createSession(JGroupHttpService.java:100)
at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:2009)
at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1945)
- locked <0x7173d1a8> (a java.util.HashMap)
at com.evermind.server.http.EvermindHttpServletRequest.unprivileged_getSession(EvermindHttpServletRequest.java:2772)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2661)
at oracle.portlet.server.containerimpl.ServerImpl.initCookie(ServerImpl.java:648)
at oracle.portlet.wsrp.v1.WSRPv1ToServer.initCookie(WSRPv1ToServer.java:6895)
at oracle.portlet.wsrp.v1.WSRP_v1_Markup_PortTypeSoapToJaxb.initCookie(WSRP_v1_Markup_PortTypeSoapToJaxb.java:624)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.invoke_initCookie(WSRP_v1_Markup_Binding_SOAP_Tie.java:362)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.processingHook(WSRP_v1_Markup_Binding_SOAP_Tie.java:785)
at oracle.j2ee.ws.server.StreamingHandler.handle(StreamingHandler.java:297)
at oracle.j2ee.ws.server.JAXRPCProcessor.doEndpointProcessing(JAXRPCProcessor.java:413)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.JAXRPCProcessor.doRequestProcessing(JAXRPCProcessor.java:277)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.JAXRPCProcessor.doService(JAXRPCProcessor.java:134)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.portlet.server.service.ContextFilter.doFilter(ContextFilter.java:86)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:623)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)

102 waiting threads on session handling

"AJPRequestHandler-HTTPThreadGroup-305" prio=10 tid=0x00d11de0 nid=0x2b3 waiting for monitor entry [0x321fe000..0x321ffbf0]
at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1944)
- waiting to lock <0x7173d1a8> (a java.util.HashMap)
at com.evermind.server.http.EvermindHttpServletRequest.unprivileged_getSession(EvermindHttpServletRequest.java:2772)

at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2661)
at oracle.portlet.server.containerimpl.ServerImpl.initCookie(ServerImpl.java:648)
at oracle.portlet.wsrp.v1.WSRPv1ToServer.initCookie(WSRPv1ToServer.java:6895)
at oracle.portlet.wsrp.v1.WSRP_v1_Markup_PortTypeSoapToJaxb.initCookie(WSRP_v1_Markup_PortTypeSoapToJaxb.java:624)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.invoke_initCookie(WSRP_v1_Markup_Binding_SOAP_Tie.java:362)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.processingHook(WSRP_v1_Markup_Binding_SOAP_Tie.java:785)
at oracle.j2ee.ws.server.StreamingHandler.handle(StreamingHandler.java:297)
at oracle.j2ee.ws.server.JAXRPCProcessor.doEndpointProcessing(JAXRPCProcessor.java:413)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.JAXRPCProcessor.doRequestProcessing(JAXRPCProcessor.java:277)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.JAXRPCProcessor.doService(JAXRPCProcessor.java:134)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

or

"AJPRequestHandler-HTTPThreadGroup-228" prio=10 tid=0x0025bc00 nid=0x266 waiting for monitor entry [0x36ffe000..0x36fffa70]
at com.evermind.server.http.JGroupHttpService.getExistingSession(JGroupHttpService.java:134)
- waiting to lock <0x7173d1a8> (a java.util.HashMap)
at com.evermind.server.http.HttpApplication.getExistingSession(HttpApplication.java:1293)

at com.evermind.server.http.EvermindHttpServletRequest.isRequestedSessionIdValid(EvermindHttpServletRequest.java:2935)
at oracle.portlet.server.containerimpl.ServerImpl.validateCookie(ServerImpl.java:1521)
at oracle.portlet.server.containerimpl.ServerImpl.getMarkup(ServerImpl.java:133)
at oracle.portlet.wsrp.v1.WSRPv1ToServer.getMarkup(WSRPv1ToServer.java:5581)
at oracle.portlet.wsrp.v1.WSRP_v1_Markup_PortTypeSoapToJaxb.getMarkup(WSRP_v1_Markup_PortTypeSoapToJaxb.java:68)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.invoke_getMarkup(WSRP_v1_Markup_Binding_SOAP_Tie.java:60)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.processingHook(WSRP_v1_Markup_Binding_SOAP_Tie.java:779)
at oracle.j2ee.ws.server.StreamingHandler.handle(StreamingHandler.java:297)
at oracle.j2ee.ws.server.JAXRPCProcessor.doEndpointProcessing(JAXRPCProcessor.java:413)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.JAXRPCProcessor.doRequestProcessing(JAXRPCProcessor.java:277)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.JAXRPCProcessor.doService(JAXRPCProcessor.java:134)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

so around createSession or getExistingSession

3) Moderate locking on object  <0x9419b378> (a com.evermind.server. http.JGroupHttpSession)

Locking thread

"AJPRequestHandler-HTTPThreadGroup-103" prio=10 tid=0x00fc8078 nid=0x183 in Object.wait() [0x3c67d000..0x3c67fbf0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:199)
- locked <0x9419b770> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.TestAndWaitLock.doAcquire(TestAndWaitLock.java:131)
at com.oracle.jgroups.TestAndWaitLock.acquire(TestAndWaitLock.java:126)
at com.oracle.bricks.j2ee.LiveSession.setLastAccessed(LiveSession.java:214)
at com.oracle.bricks.j2ee.LiveSession.get(LiveSession.java:296)
at com.oracle.bricks.j2ee.SessionFacade.get(SessionFacade.java:64)
at com.evermind.server.http.JGroupHttpSession.getValueFromMap(JGroupHttpSession.java:138)
at com.evermind.server.http.EvermindHttpSession.getAttribute(EvermindHttpSession.java:209)
- locked <0x9419b378> (a com.evermind.server.http.JGroupHttpSession)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSessionInternal(EntityReference.java:444)
- locked <0x9419b378> (a com.evermind.server.http.JGroupHttpSession)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSession(EntityReference.java:317)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSessionHandle(EntityReference.java:421)
at oracle.portlet.server.containerimpl.PortletResponseImpl.encodeURL(PortletResponseImpl.java:150)
at oracle.portlet.server.containerimpl.RenderResponseImpl.encodeURL(RenderResponseImpl.java:704)
at _twProposalForm._jspService(_twProposalForm.java:365)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:462)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

10 waiting threads on this object,
patter of waiting threads is

"AJPRequestHandler-HTTPThreadGroup-102" prio=10 tid=0x007cc870 nid=0x17c waiting for monitor entry [0x3d0fe000..0x3d0ffb70]
at com.evermind.server.http.EvermindHttpSession.beginRequest(EvermindHttpSession.java:1006)
- waiting to lock <0x9419b378> (a com.evermind.server.http.JGroupHttpSession)
at com.evermind.server.http.JGroupHttpSession.beginRequest(JGroupHttpSession.java:178)

at com.evermind.server.http.EvermindHttpServletRequest.unprivileged_getSession(EvermindHttpServletRequest.java:2778)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2661)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSessionInternal(EntityReference.java:436)
at oracle.portlet.server.containerimpl.EntityReference. (EntityReference.java:127)
at oracle.portlet.server.containerimpl.PortletEntity.getEntityReference(PortletEntity.java:133)
at oracle.portlet.server.containerimpl.ServerImpl.performBlockingInteraction(ServerImpl.java:482)
at oracle.portlet.wsrp.v1.WSRPv1ToServer.performBlockingInteraction(WSRPv1ToServer.java:6538)
at oracle.portlet.wsrp.v1.WSRP_v1_Markup_PortTypeSoapToJaxb.performBlockingInteraction(WSRP_v1_Markup_PortTypeSoapToJaxb.java:301)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.invoke_performBlockingInteraction(WSRP_v1_Markup_Binding_SOAP_Tie.java:447)
at oasis.names.tc.wsrp.v1.bind.runtime.WSRP_v1_Markup_Binding_SOAP_Tie.processingHook(WSRP_v1_Markup_Binding_SOAP_Tie.java:788)
at oracle.j2ee.ws.server.StreamingHandler.handle(StreamingHandler.java:297)
at oracle.j2ee.ws.server.JAXRPCProcessor.doEndpointProcessing(JAXRPCProcessor.java:413)
at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
at oracle.j2ee.ws.server.JAXRPCProcessor.doRequestProcessing(JAXRPCProcessor.java:277)
at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
at oracle.j2ee.ws.server.JAXRPCProcessor.doService(JAXRPCProcessor.java:134)
at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.portlet.server.service.ContextFilter.doFilter(ContextFilter.java:86)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:623)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) at java.lang.Thread.run(Thread.java:595) Threads from Node2 

Contention appears in 3 objects.. almost all them with the very same kind of objects

"AJPRequestHandler-HTTPThreadGroup-19" prio=10 tid=0x0120e358 nid=0x119 in Object.wait() [0x403fd000..0x403ff8f0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474)
at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:199)
- locked <0x724eede8> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
at com.oracle.jgroups.TestAndWaitLock.doAcquire(TestAndWaitLock.java:131)
at com.oracle.jgroups.TestAndWaitLock.acquire(TestAndWaitLock.java:126)
at com.oracle.bricks.j2ee.LiveSession.setLastAccessed(LiveSession.java:214)
at com.oracle.bricks.j2ee.LiveSession.get(LiveSession.java:296)
at com.oracle.bricks.j2ee.SessionFacade.get(SessionFacade.java:64)
at com.evermind.server.http.JGroupHttpSession.getValueFromMap(JGroupHttpSession.java:138)
at com.evermind.server.http.EvermindHttpSession.getAttribute(EvermindHttpSession.java:209)
- locked <0x724eebc0> (a com.evermind.server.http.JGroupHttpSession)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSessionInternal(EntityReference.java:444)
- locked <0x724eebc0> (a com.evermind.server.http.JGroupHttpSession)
at oracle.portlet.server.containerimpl.EntityReference.getPortletSession(EntityReference.java:317)
at oracle.portlet.server.containerimpl.PortletRequestImpl.getPortletSession(PortletRequestImpl.java:316)
at oracle.portlet.server.containerimpl.PortletRequestImpl.getPortletSession(PortletRequestImpl.java:299)
at _refNumberSearch._jspService(_refNumberSearch.java:89)

However in this case, Node2 is not displaying the same level of contention => 209 threads with 15% of all threads are waiting for a monitor

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms