Oc4j Cluster Hangs & Oc4j Restarts With Or Without Load Due to Unresponsiveness (Doc ID 1422043.1)

Last updated on DECEMBER 05, 2016

Applies to:

Oracle Containers for J2EE - Version 10.1.3.5.0 and later
Information in this document applies to any platform.
***Checked for relevance on 11-Nov-2013***

Symptoms


In an Active-Passive AS 10.1.3.5 cluster, OC4J instances appear to be in hang status.
OPMN reports it has to restart the OC4J instances due to unresponsiveness

After taking thread dumps of the hung JVM , we could observe big contention in the session replication involved threads as follows:
Locked object 0xce7fd1d8 com.evermind.server.http.JGroupHttpSession 

Locking thread is

"AJPRequestHandler-RMICallHandler-58" prio=10 tid=0xaa9a0000 nid=0x53a2 in Object.wait() [0xa9790000]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb3adc208> (a java.lang.Object)
at org.jgroups.protocols.FC.handleDownMessage(FC.java:360)
- locked <0xb3adc208> (a java.lang.Object)
at org.jgroups.protocols.FC.down(FC.java:300)
at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
at org.jgroups.protocols.FC.receiveDownEvent(FC.java:294)
at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
at org.jgroups.protocols.FRAG.down(FRAG.java:139)
at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
at org.jgroups.JChannel.down(JChannel.java:1115)
at org.jgroups.JChannel.send(JChannel.java:588)
at com.oracle.jgroups.ChannelAdapter.broadcast(ChannelAdapter.java:461)
at com.oracle.bricks.LocalObject.send(LocalObject.java:223)
at com.oracle.bricks.j2ee.LiveSession.flushMap(LiveSession.java:515)
at com.oracle.bricks.j2ee.SessionFacade.flushMap(SessionFacade.java:102)
at com.evermind.server.http.JGroupHttpSession.flush(JGroupHttpSession.java:144)
at com.evermind.server.http.JGroupHttpSession.putValueToMap(JGroupHttpSession.java:135)
at com.evermind.server.http.EvermindHttpSession.setAttribute(EvermindHttpSession.java:171)
- locked <0xce7fd1d8> (a com.evermind.server.http.JGroupHttpSession)
at com.evermind.server.http.EvermindHttpSession.setAttribute(EvermindHttpSession.java:137)
at com.intrum.fpw.servlet.CaseServlet.doGet(CaseServlet.java:151)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at com.intrum.fpw.servlet.FrameworkServlet.service(FrameworkServlet.java:310)
at com.intrum.fpw.servlet.FrameworkServlet.service(FrameworkServlet.java:364)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at com.intrum.fpw.servlet.filter.Log4jNDCFilter.doFilter(Log4jNDCFilter.java:119)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.doFilter(UserContextSecurityFilter.java:40)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.AntiSamyFilter.doFilter(AntiSamyFilter.java:29)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.EncodingFilter.doFilter(EncodingFilter.java:51)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)

11 waiting threads on that lock object

"AJPRequestHandler-RMICallHandler-72" prio=10 tid=0x08a46000 nid=0x40cd waiting for monitor entry [0xab542000]
  java.lang.Thread.State: BLOCKED (on object monitor)
at com.evermind.server.http.EvermindHttpSession.beginRequest(EvermindHttpSession.java:1025)
- waiting to lock <0xce7fd1d8> (a com.evermind.server.http.JGroupHttpSession)
at com.evermind.server.http.JGroupHttpSession.beginRequest(JGroupHttpSession.java:203)
at com.evermind.server.http.EvermindHttpServletRequest.unprivileged_getSession(EvermindHttpServletRequest.java:2839)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2714)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2709)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2704)
at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:268)
at com.intrum.fpw.web.HTMLUserInterfaceHelper.getCurrentUser(HTMLUserInterfaceHelper.java:531)
at com.intrum.fpw.servlet.FrameworkServlet.getFrameworkUser(FrameworkServlet.java:545)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.createUserContext(UserContextSecurityFilter.java:68)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.doFilter(UserContextSecurityFilter.java:36)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.AntiSamyFilter.doFilter(AntiSamyFilter.java:29)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.EncodingFilter.doFilter(EncodingFilter.java:51)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms