My Oracle Support Banner

Oc4j Cluster Hangs & Oc4j Restarts With Or Without Load Due to Unresponsiveness (Doc ID 1422043.1)

Last updated on JUNE 22, 2018

Applies to:

Oracle Containers for J2EE - Version 10.1.3.5.0 and later
Information in this document applies to any platform.

Symptoms


In an Active-Passive AS 10.1.3.5 cluster, OC4J instances appear to be in hang status.
OPMN reports it has to restart the OC4J instances due to unresponsiveness

After taking thread dumps of the hung JVM , we could observe big contention in the session replication involved threads as follows:
Locked object 0xce7fd1d8 com.evermind.server.http.JGroupHttpSession 

Locking thread is

"AJPRequestHandler-RMICallHandler-58" prio=10 tid=0xaa9a0000 nid=0x53a2 in Object.wait() [0xa9790000]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb3adc208> (a java.lang.Object)
at org.jgroups.protocols.FC.handleDownMessage(FC.java:360)
- locked <0xb3adc208> (a java.lang.Object)
at org.jgroups.protocols.FC.down(FC.java:300)
at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
at org.jgroups.protocols.FC.receiveDownEvent(FC.java:294)
at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
at org.jgroups.protocols.FRAG.down(FRAG.java:139)
at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
at org.jgroups.JChannel.down(JChannel.java:1115)
at org.jgroups.JChannel.send(JChannel.java:588)
at com.oracle.jgroups.ChannelAdapter.broadcast(ChannelAdapter.java:461)
at com.oracle.bricks.LocalObject.send(LocalObject.java:223)
at com.oracle.bricks.j2ee.LiveSession.flushMap(LiveSession.java:515)
at com.oracle.bricks.j2ee.SessionFacade.flushMap(SessionFacade.java:102)
at com.evermind.server.http.JGroupHttpSession.flush(JGroupHttpSession.java:144)
at com.evermind.server.http.JGroupHttpSession.putValueToMap(JGroupHttpSession.java:135)
at com.evermind.server.http.EvermindHttpSession.setAttribute(EvermindHttpSession.java:171)
- locked <0xce7fd1d8> (a com.evermind.server.http.JGroupHttpSession)
at com.evermind.server.http.EvermindHttpSession.setAttribute(EvermindHttpSession.java:137)
at com.intrum.fpw.servlet.CaseServlet.doGet(CaseServlet.java:151)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at com.intrum.fpw.servlet.FrameworkServlet.service(FrameworkServlet.java:310)
at com.intrum.fpw.servlet.FrameworkServlet.service(FrameworkServlet.java:364)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at com.intrum.fpw.servlet.filter.Log4jNDCFilter.doFilter(Log4jNDCFilter.java:119)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.doFilter(UserContextSecurityFilter.java:40)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.AntiSamyFilter.doFilter(AntiSamyFilter.java:29)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.EncodingFilter.doFilter(EncodingFilter.java:51)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)

11 waiting threads on that lock object

"AJPRequestHandler-RMICallHandler-72" prio=10 tid=0x08a46000 nid=0x40cd waiting for monitor entry [0xab542000]
  java.lang.Thread.State: BLOCKED (on object monitor)
at com.evermind.server.http.EvermindHttpSession.beginRequest(EvermindHttpSession.java:1025)
- waiting to lock <0xce7fd1d8> (a com.evermind.server.http.JGroupHttpSession)
at com.evermind.server.http.JGroupHttpSession.beginRequest(JGroupHttpSession.java:203)
at com.evermind.server.http.EvermindHttpServletRequest.unprivileged_getSession(EvermindHttpServletRequest.java:2839)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2714)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2709)
at com.evermind.server.http.EvermindHttpServletRequest.getSession(EvermindHttpServletRequest.java:2704)
at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:268)
at com.intrum.fpw.web.HTMLUserInterfaceHelper.getCurrentUser(HTMLUserInterfaceHelper.java:531)
at com.intrum.fpw.servlet.FrameworkServlet.getFrameworkUser(FrameworkServlet.java:545)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.createUserContext(UserContextSecurityFilter.java:68)
at com.intrum.fpw.servlet.filter.UserContextSecurityFilter.doFilter(UserContextSecurityFilter.java:36)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.AntiSamyFilter.doFilter(AntiSamyFilter.java:29)
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
at com.intrum.fpw.servlet.filter.EncodingFilter.doFilter(EncodingFilter.java:51)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:642)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:391)
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:908)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.