JOC Deadlocks caused by oracle.ias.cache.group.GrpChannel.multicast in E-Business Suite R12 (Doc ID 730492.1)

Last updated on JULY 28, 2016

Applies to:

Oracle EBS Applications Performance - Version 12.0.0 to 12.1.1 [Release 12.0 to 12.1]
Information in this document applies to any platform.
This problem can occur on any platform.

Symptoms

On Oracle Applications R12.0.x some customers will face the following performance problem:

Symptoms:

1- Rapid increase in the number of JDBC sessions on the server
2- OACore service is hanging
3- End users can not login to Oracle Applications
4- Existing and connected users can not navigate through framework pages and receive the following message

    Error:
    500 - Internal Server Error

What needs to be checked:

1- Generate java thread dump for the hanging JVM and check the JVM log file to see if there is deadlock on the following object (oracle.ias.cache.group.GrpChannel.multicast).

You can use command like this to generate the java thread dump:

$ kill -3 <jvm pid>

sample output in the thread dump log will show locking session like the following:

[ Locking Thread ] 
"Worker-4" daemon prio=1 tid=0x088cd2e8 nid=0x2211 runnable 
[0x9683c000..0x9683cfb0] 
at java.net.SocketOutputStream.socketWrite0(Native Method) 
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) 
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) 
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) 
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) 
- locked <0xb1c31b60> (a java.io.BufferedOutputStream) 
at oracle.ias.cache.group.HttpHandler.write(Unknown Source) 
at oracle.ias.cache.group.StreamHandler.write(Unknown Source) 
at oracle.ias.cache.group.EndPoint.write(Unknown Source) 
at oracle.ias.cache.group.Transport.syncMulticast(Unknown Source) 
- locked <0xb1c31a08> (a [B) 
at oracle.ias.cache.group.Transport.multicast(Unknown Source) 
at oracle.ias.cache.group.Transport.multicast(Unknown Source) 
at oracle.ias.cache.group.GrpChannel.multicast(Unknown Source) 
- locked <0xb05e7380> (a [B) 
at oracle.ias.cache.group.GrpChannel.multicast(Unknown Source) 
at oracle.ias.cache.Net.broadcast(Unknown Source) 
at oracle.ias.cache.Net.broadcast(Unknown Source) 
at oracle.ias.cache.CacheHandle.broadcastManipulation(Unknown Source) 
at oracle.ias.cache.CacheHandle.manipulateCascade(Unknown Source) 
at oracle.ias.cache.CacheHandle.manipulate(Unknown Source) 
at oracle.ias.cache.CacheHandle.invalidate(Unknown Source) 
at oracle.ias.cache.Task.execute(Unknown Source) 
at oracle.ias.cache.WorkerThread.run(Unknown Source) 

[ Waiting Thread ] ( one example from 160 similar threads waiting for lock monitor ) 
"AJPRequestHandler-ApplicationServerThread-161" prio=1 tid=0x9ecf1d80 
nid=0x4070 waiting for monitor entry [0x93309000..0x93309eb0] 
at oracle.ias.cache.group.GrpChannel.multicast(Unknown Source) 
- waiting to lock <0xb05e7380> (a [B) 
at oracle.ias.cache.group.GrpChannel.multicast(Unknown Source) 
at oracle.ias.cache.Net.broadcast(Unknown Source) 
at oracle.ias.cache.Net.broadcast(Unknown Source) 
at oracle.ias.cache.CacheHandle.broadcastManipulation(Unknown Source) 
at oracle.ias.cache.CacheHandle.manipulateCascade(Unknown Source) 
at oracle.ias.cache.CacheHandle.manipulate(Unknown Source) 
at oracle.ias.cache.CacheHandle.invalidate(Unknown Source) 
at oracle.ias.cache.CacheHandle.checkTtl(Unknown Source) 
at oracle.ias.cache.CacheHandle.findObject(Unknown Source) 
at oracle.ias.cache.CacheHandle.locateObject(Unknown Source) 
at oracle.ias.cache.CacheAccess.get(Unknown Source) 
at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:650) 
at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802) 
at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624) 
at oracle.apps.fnd.cache.AppsCache.get(AppsCache.java:216) 
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1485) 
at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:354) 
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(ExtendedProfileStore.java:211) 
at oracle.apps.fnd.common.NativeProfileStore.getMultiSpecificProfileFromDB(Native ProfileStore.java:589) 
at oracle.apps.fnd.common.WebAppsContext.setProfileValues(WebAppsContext.java:4769) 
at oracle.apps.fnd.common.AppsContext.setDBEnv(AppsContext.java:3494) 
at oracle.apps.fnd.common.AppsContext.getPrivateConnectionFinal(AppsContext.java: 2595)
at oracle.apps.fnd.common.AppsContext.getPrivateConnection(AppsContext.java:2485)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2344) 
- locked <0xdd65feb8> (a oracle.apps.fnd.common.WebAppsContext) 
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2159) 
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2063) 
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2080) 
at oracle.apps.fnd.common.Context.getJDBCConnection(Context.java:1617) 
at oracle.apps.fnd.sso.Utils.getConnection(Utils.java:625) 
at oracle.apps.fnd.sso.AppsLoginRedirect.doGet(AppsLoginRedirect.java:520) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) 
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64) 
at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26) 
at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15) 
at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:318) 
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:610) 
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:359) 
at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:870) 
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:451) 
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:299) 
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:187) 
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260) 
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:230) 
at oracle.oc4j.network.ServerSocketAcceptHandler.access$800(ServerSocketAcceptHandler.java:33) 
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:831) 
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303) 
at java.lang.Thread.run(Thread.java:595)

2- While the issue is occuring and deadlock detected from step 1 you may see following error message in the application log file for OACore group:

java.lang.OutOfMemoryError: Java heap space

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms