OIM Going into Warning State and WebLogic Console Showing Stuck Threads [waiting on (a com.sun.jndi.ldap.LdapRequest)] (Doc ID 1923106.1)

Last updated on FEBRUARY 13, 2017

Applies to:

Identity Manager - Version 11.1.2.0.0 and later
Oracle Internet Directory - Version 10.1.4 to 11.1.1 [Release 10gR3 to 11g]
Information in this document applies to any platform.

Symptoms

Oracle Identity Manager (OIM) 11g integrated with Oracle Internet Directory (OID) 10g or 11g.  No OIM ldap sync and no libovd are involved, only a custom OIM LDAP connector for OID is being used.

OIM going into warning state and Weblogic console is showing stuck threads.

Depending on what server in OIM weblogic cluster is running jobs, seeing different 'STUCK' messages in the oim_server diagnostic logs along with the health of the OIM servers going into a 'Warning' state in the Weblogic console. These stuck threads are causing downstream effects on the jobs running.

OIM log does not show many threads for OIM, however it shows almost 2000 LDAPConnectionThread's, for example:

"DataChange - 504962" daemon prio=10 tid=0x00007f3bc42e9800 nid=0x2a74 in
Object.wait() [0x00007f3b7d2cf000]
  java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a com.sun.jndi.ldap.LdapRequest)
at com.sun.jndi.ldap.Connection.readReply(Connection.java:464)
- locked (a com.sun.jndi.ldap.LdapRequest)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:631)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:554)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)


OIM logs may also show:

[2014-07-28T12:00:00.410-04:00] [oim_server] [ERROR] [] [] [tid: DataChange - 564524] [..] javax.naming.NamingException: LDAP response read timed out, timeout used:600000ms.; remaining name 'dc=mycompany,dc=com'[[
javax.naming.NamingException: LDAP response read timed out, timeout used:600000ms.; remaining name 'dc=mycompany,dc=com'
at com.sun.jndi.ldap.Connection.readReply(Connection.java:481)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:631)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:554)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
...<etc,etc>...
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5170)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

]]
[2014-07-28T12:00:00.412-04:00] [oim_server3] [TRACE] [] [] [tid: DataChange - 564524] [userId: oiminternal] [..] [SRC_CLASS: com.identityworksllc.queue.ConnectionPool$PoolEntry] [SRC_METHOD: getConnection:105] Connection com.identityworksllc.integrations.ldap.PooledDirContext@1d6f56c8 failed isWorking() check;
re-creating...
[2014-07-28T12:00:00.438-04:00] [oim_server3] [ERROR] [] [com.mycompany.OIM.INTEGRATIONS.mycompanyOIDINTEGRATION] [tid: DataChange - 564525] [userId: oiminternal] [..] javax.naming.NamingException: LDAP response read timed out, timeout used:600000ms.; remaining name 'dc=mycompany,dc=com'[[
javax.naming.NamingException: LDAP response read timed out, timeout used:600000ms.; remaining name 'dc=mycompany,dc=com'
at com.sun.jndi.ldap.Connection.readReply(Connection.java:481)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:631)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:554)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)



No OID log errors can be found that correspond to date/time of failures, and no high OID operation times (OPtime) values in debugged OID logs either.


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms