My Oracle Support Banner

OID 12c ldap Processes Shutting Down and Not Respawning or Hanging. Log Error Includes: Recieved signal = 11, shutting down (Doc ID 2820111.1)

Last updated on NOVEMBER 22, 2021

Applies to:

Oracle Internet Directory - Version 12.2.1.3.0 and later
Information in this document applies to any platform.

Symptoms

Oracle Internet Directory (OID) ldap processes shutting down and not re-spawning / not being restarted by oidmon, or hanging.

OID ldap server log errors include:

[2020-03-17T18:42:12.28353+00:00] [OID] [NOTIFICATION:16] [] [OIDLDAPD] [host: <HOSTNAME>] [pid: <PID>] [tid: <TID>] gslarpsbPsearchListener: sgslusccConnectPort:connect (AF_UNIX, path=./.oidldapd.<PID>) failed,OS error = (2)

And/Or,

Recieved signal = 11, shutting down

Or,

Recieved signal = 6, shutting down

 


Stack dump logs (*stack*.dmp) may include:

Dumping stack trace for OIDLDAPD (server) ...
calling               caller                argument values in hex
location                                     (? means dubious value)
--------------------------------------------------------------------
gslusdsDumpStack()  __sighandler()           arg#1: ???? value=00000000 (ptr)
                                             arg#2: ???? value=00000000 (ptr)
. . .
__sighandler()      gsignal()                arg#1: ???? value=00005A5A (ptr)
                                             arg#2: ???? value=00005D00 (ptr)
. . .
gsignal()           _int_free()              arg#1: ???? value=00000000 (ptr)
                                             arg#2: ???? value=00000000 (ptr)

. . .
< Dump of dispatcher connections >

 

After dispatcher thread, the file will either end or continue as below:

Dumping connection stat for dispather thread : <number>
*** Error in `oidldapd': corrupted double-linked list: 0x00002b9f1902e5e0 ***
    <=  NOTE
======= Backtrace: =========
/lib64/libc.so.6(+0x80adf)[0x2b9e58538adf]
/lib64/libc.so.6(+0x8136e)[0x2b9e5853936e]
oidldapd[0x41d064]
oidldapd[0x41cfe9]
oidldapd(gslumfFree+0x74)[0x42cb34]
oidldapd[0x4b69dc]
oidldapd[0x50a5f3]
oidldapd[0x4bfaab]
/lib64/libpthread.so.0(+0x7dd5)[0x2b9e5244edd5]
/lib64/libc.so.6(clone+0x6d)[0x2b9e585b5ead]
======= Memory map: ========
00400000-006fe000 r-xp 00000000 fd:09 152688178
<FMWDIR>/bin/oidldapd
008fd000-008fe000 r--p 002fd000 fd:09 152688178
<FMWDIR>/bin/oidldapd
008fe000-0090a000 rw-p 002fe000 fd:09 152688178
<FMWDIR>/bin/oidldapd
0090a000-0090d000 rw-p 00000000 00:00 0
0237a000-41d12000 rw-p 00000000 00:00 0
[heap]
2b8cfe5d4000-2b8d4a5d6000 rw-s 00000000 00:04 1030750211
/SYSV00007bd3 (deleted)         <= NOTE
2b9e51d8c000-2b9e51dae000 r-xp 00000000 fd:00 37758220
/usr/lib64/ld-2.17.so
...



After one or more process failures, the OID instance may hang entirely. Below is an example stack from the dispatcher processes during an OID hang:

...
Thread 4 (Thread 0x2af311452700 (LWP 5635)):
#0  0x00000000005f083f in gslarddDispatcher ()
#1  0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#2  0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2af311c56700 (LWP 5748)):
#0  0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4  0x00000000025bab3f in sltsmna ()
#5  0x0000000000568b5d in gslupmaMutexAcquire ()
#6  0x00000000005f3ab9 in gslardlGetServer ()
#7  0x00000000005f3845 in gslardlAcceptConnAndSend ()
#8  0x00000000005f35f9 in gslardlAcceptConns ()
#9  0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2af312c5e700 (LWP 5756)):
#0  0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4  0x00000000025bab3f in sltsmna ()
#5  0x0000000000568b5d in gslupmaMutexAcquire ()
#6  0x00000000005f3ab9 in gslardlGetServer ()
#7  0x00000000005f3845 in gslardlAcceptConnAndSend ()
#8  0x00000000005f35f9 in gslardlAcceptConns ()
#9  0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2af306e7cf80 (LWP 4433)):
#0  0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4  0x00000000025bab3f in sltsmna ()
#5  0x0000000000568b5d in gslupmaMutexAcquire ()
#6  0x000000000062a820 in gslardcController ()
#7  0x00000000005ea8c5 in gslarmcController ()
#8  0x000000000054cdf9 in gslmdispMain ()
#9  0x000000000054bf76 in main ()
...

 

May also see /var/log/messages such as below while grepping for kill:

...
Apr 17 07:32:32 <hostname> abrt-hook-ccpp: Process 7566 (frmweb) of user 1151 killed by SIGSEGV - dumping core
Apr 17 08:03:55 <hostname>  abrt-hook-ccpp: Process 26218 (oidldapd) of user 1165 killed by SIGSEGV - ignoring (repeated crash)
Apr 17 08:03:55 <hostname>  abrt-hook-ccpp: Process 26308 (oidldapd) of user 1165 killed by SIGSEGV - dumping core
Apr 17 08:04:08 <hostname>  abrt-hook-ccpp: Process 26243 (oidldapd) of user 1165 killed by SIGSEGV - ignoring (repeated crash)
Apr 17 08:32:04 <hostname>  abrt-hook-ccpp: Process 6746 (oidldapd) of user 1165 killed by SIGABRT - dumping core
Apr 17 09:31:40 <hostname>  abrt-hook-ccpp: Process 10584 (frmweb) of user 1151 killed by SIGSEGV - dumping core
Apr 17 10:00:20 <hostname>  abrt-hook-ccpp: Process 21899 (oidldapd) of user 1165 killed by SIGSEGV - dumping core
...

The dates/times match the stack dumps leading to a respawn.

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.