OID 11g Takes a Long Time (Hours) to Start and then Does Not Respond: Can't Contact LDAP Server (Doc ID 2104886.1)

Last updated on JUNE 30, 2017

Applies to:

Oracle Internet Directory - Version 11.1.1 and later
Information in this document applies to any platform.

Symptoms

Oracle Internet Directory (OID) 11g, e.g., 11.1.1.6.0 version.

There is no load balancer or firewall in between OID and its Database (DB).

The groups are not too large, but there are over 135K groups, and over 2000 of them are privilege groups.

The oid is not starting when using opmnctl startall:

$ opmnctl startall
opmnctl startall: starting opmn and all managed processes...
================================================================================
opmn id=myoidhost.mycompany.com:6701
Response: 2 of 3 processes started.

ias-instance id=oid_prod_instance
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
  oid1/oidmon/OID/

Error
--> Process (index=1,uid=1404495441,pid=5140)
  time out while waiting for a managed process to start
  Log:
  /u01/products/middleware/11g/oid/diagnostics/logs/OID/oid1/console~OID~1.log


It takes around 1-2 hours to bring up the OID processes, i.e., for it to return the prompt back from an oidctl command. And then OID does not respond to ldapbinds; it returns "can't contact ldap server" error.

 

OID dispatcher oidldapd01-0000.log shows:

[2016-01-20T11:44:31-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: ERROR : gslsflAcceptConnAndSend : No serever process availbale to dispatch TCP Connection

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: sgslun6xAccept: Accept succedded on new communication endpoint

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: INFO : gslsflgGetServer : Entry

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: INFO: gslsflgGetServer : Exit
[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: ERROR : gslsflAcceptConnAndSend : No serever process availbale to dispatch TCP Connection

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: sgslun6xAccept: Accept succedded on new communication endpoint

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: INFO : gslsflgGetServer : Entry

[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: INFO: gslsflgGetServer : Exit
[2016-01-20T11:47:14-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 12212] [tid: 2] DispatcherListener: ERROR : gslsflAcceptConnAndSend : No serever process availbale to dispatch TCP Connection

 

OID server logs may show lots of group activities such as:

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Subtree Search Completed with: 100

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: [gsldfd_FetchSubTreeObjs]Re-using prepared Statement

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Bind variable: :0 = "cn=group1,cn=groups,dc=mycompany,dc=com"

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Bind variable: :1 = "cn=group1,cn=groups,dc=mycompany,dc=com"

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Subtree Search Completed with: 100

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: [gsldfd_FetchSubTreeObjs]Re-using prepared Statement

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Bind variable: :0 = "cn=group2,cn=groups,dc=mycompany,dc=com"

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Bind variable: :1 = "cn=group2,cn=groups,dc=mycompany,dc=com"

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: Subtree Search Completed with: 100

[2016-01-20T01:00:44-05:00] [OID] [TRACE:16] [] [OIDLDAPD] [host: myoidhost.mycompany.com] [pid: 2370] [tid: 0] Main:: [gsldfd_FetchSubTreeObjs]Re-using prepared Statement

 

OID stack files may be generated, for example:

Dumping stack trace for OIDLDAPD (server) ...
calling caller argument values in hex
location (? means dubious value)
--------------------------------------------------------------------
gslusdsDumpStack() __restore_rt() arg#1: ???? value=C1899C38 (ptr)
arg#2: ???? value=C1896010 (ptr)
arg#3: ???? value=C1897528 (ptr)
arg#4: ???? value=C18975F0 (ptr)
arg#5: ???? value=C1899C38 (ptr)
arg#6: ???? value=00000001 (ptr)

__restore_rt() read()+16 arg#1: ???? value=00000014 (ptr)
arg#2: ???? value=0147EA06 (ptr)
arg#3: ???? value=00002010 (ptr)
arg#4: ???? value=FFFFFFFF (ptr)
arg#5: ???? value=00000008 (ptr)
arg#6: ???? value=00000000 (ptr)

read() snttread()+14 arg#1: ???? value=00000014 (ptr)
arg#2: ???? value=0147EA06 (ptr)
arg#3: ???? value=00002010 (ptr)
arg#4: ???? value=FFFFFFFF (ptr)
arg#5: ???? value=00000008 (ptr)
arg#6: ???? value=00000000 (ptr)

snttread() nttfprd()+157 arg#1: ???? value=00000014 (ptr)
arg#2: ???? value=0147EA06 (ptr)
arg#3: ???? value=00002010 (ptr)
arg#4: ???? value=FFFFFFFF (ptr)
arg#5: ???? value=00000008 (ptr)
arg#6: ???? value=00000000 (ptr)

...<etc>...

 


Tried disabling group cache, and all cache, but the same problem remains.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms