OIA Application Going Down Due To Stuck Threads Exceptions

(Doc ID 1576506.1)

Last updated on NOVEMBER 09, 2016

Applies to:

Oracle Identity Analytics - Version 11.1.1.3.0 and later
Information in this document applies to any platform.

Symptoms

This can be reproduced at will by the following:

Log on to OIA and from the home screen click on Identity Certification then Dashboard ( referred to as 1) and then back to Home (referred to as 2) (upper right corner) then again to (1) and then again to (2) and again to (1) and again to (2) etc. etc. etc.  No pauses in-between.

In testing, we also found that having a lot of certification data is related.  For example, IDC_ACCOUNT_ATTRIBUTES table is about 80G in size.
That makes the query take about 18 minutes.
In that amount of time a lot of database sessions could be open.

Here is an example of the problem query:

SELECT COUNT(A.ID) AS TOTAL_ACCOUNTS
FROM IDC_ACCOUNTS A
WHERE EXISTS (
SELECT UA.ACCOUNT_ID
FROM IDC_USER_ACCOUNTS UA
WHERE A.ID = UA.ACCOUNT_ID
AND UA.CERT_ID IN (...)
UNION SELECT VU.ACCOUNT_ID
FROM ID_ATTR_VAL_USERS VU
WHERE  A.ID = VU.ACCOUNT_ID
AND VU.CERT_ID IN (...))

 

When the behavior appears, we see the following:

OIA LOGS:

07:23:47,106 DEBUG [AuthenticationEventsListener] Listening application event
07:23:47,106 DEBUG [DefaultListableBeanFactory] Returning cached instance of singleton bean 'versionService'
07:23:47,107 INFO [ContextLifecycleListener] Oracle Identity Analytics (build: 11.1.1.3.1.20120615_I_11791) Started

===
Weblogic logs

POST /rbacx/secure/dwr/call/plaincall/dwrIDCService.generateCertificationChartByCertifyRevokeCounts.dwr HTTP/1.1
Accept: */*
Accept-Language: en-us
Referer: http://168.108.227.72:8001/rbacx/secure/home/home.action
Content-Type: text/plain
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; MS-RTC LM 8)
Content-Length: 284
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: JSESSIONID=0yXWRTTMsLhGcN1vvVvpyrw6GmsH9fRwdky4PyfyQQW2GggRSj5x!1699652099

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:92)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:589)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3386)
oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3487)
oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)
weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:98)
sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.ibatis.common.jdbc.logging.PreparedStatementLogProxy.invoke(PreparedStatementLogProxy.java:62)
$Proxy148.execute(Unknown Source)
com.ibatis.sqlmap.engine.execution.DefaultSqlExecutor.executeQuery(DefaultSqlExecutor.java:183)
com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)
com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForObject(MappedStatement.java:120)
com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:529)
com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:504)
com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:106)
org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:209)
org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(SqlMapClientTemplate.java:271)
com.vaau.rbacx.dao.ibatis.SqlMapRbacxStatisticsDao.getRbacxStatistics(SqlMapRbacxStatisticsDao.java:60)
com.vaau.rbacx.service.impl.RbacxServiceImpl.getRbacxStatistics(RbacxServiceImpl.java:1178)
sun.reflect.GeneratedMethodAccessor858.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
$Proxy343.getRbacxStatistics(Unknown Source)
sun.reflect.GeneratedMethodAccessor858.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:113)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy344.getRbacxStatistics(Unknown Source)
sun.reflect.GeneratedMethodAccessor858.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
$Proxy345.getRbacxStatistics(Unknown Source)
sun.reflect.GeneratedMethodAccessor858.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy345.getRbacxStatistics(Unknown Source)
sun.reflect.GeneratedMethodAccessor858.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
$Proxy345.getRbacxStatistics(Unknown Source)
com.vaau.rbacx.idc.web.dwr.impl.DwrIDCServiceImpl.generateCertificationChartByCertifyRevokeCounts(DwrIDCServiceImpl.java:1715)
sun.reflect.GeneratedMethodAccessor1606.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy369.generateCertificationChartByCertifyRevokeCounts(Unknown Source)
sun.reflect.GeneratedMethodAccessor1605.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.directwebremoting.impl.ExecuteAjaxFilter.doFilter(ExecuteAjaxFilter.java:34)
org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:431)
org.directwebremoting.impl.DefaultRemoter.execute(DefaultRemoter.java:283)
org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:52)
org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)
org.directwebremoting.spring.DwrController.handleRequestInternal(DwrController.java:234)
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms