OID 12c ldap Processes Shutting Down and Not Respawning or Hanging. Log Error Includes: Recieved signal = 11, shutting down
(Doc ID 2820111.1)
Last updated on FEBRUARY 01, 2024
Applies to:
Oracle Internet Directory - Version 12.2.1.3.0 and laterInformation in this document applies to any platform.
Symptoms
Oracle Internet Directory (OID) ldap processes shutting down and not re-spawning / not being restarted by oidmon, or hanging.
OID ldap server log errors include:
And/Or,
Or,
Stack dump logs (*stack*.dmp) may include:
Dumping stack trace for OIDLDAPD (server) ...
calling caller argument values in hex
location (? means dubious value)
--------------------------------------------------------------------
gslusdsDumpStack() __sighandler() arg#1: ???? value=00000000 (ptr)
arg#2: ???? value=00000000 (ptr)
. . .
__sighandler() gsignal() arg#1: ???? value=00005A5A (ptr)
arg#2: ???? value=00005D00 (ptr)
. . .
gsignal() _int_free() arg#1: ???? value=00000000 (ptr)
arg#2: ???? value=00000000 (ptr)
. . .
< Dump of dispatcher connections >
After dispatcher thread, the file will either end or continue as below:
*** Error in `oidldapd': corrupted double-linked list: 0x00002b9f1902e5e0 ***
<= NOTE
======= Backtrace: =========
/lib64/libc.so.6(+0x80adf)[0x2b9e58538adf]
/lib64/libc.so.6(+0x8136e)[0x2b9e5853936e]
oidldapd[0x41d064]
oidldapd[0x41cfe9]
oidldapd(gslumfFree+0x74)[0x42cb34]
oidldapd[0x4b69dc]
oidldapd[0x50a5f3]
oidldapd[0x4bfaab]
/lib64/libpthread.so.0(+0x7dd5)[0x2b9e5244edd5]
/lib64/libc.so.6(clone+0x6d)[0x2b9e585b5ead]
======= Memory map: ========
00400000-006fe000 r-xp 00000000 fd:09 152688178
<FMWDIR>/bin/oidldapd
008fd000-008fe000 r--p 002fd000 fd:09 152688178
<FMWDIR>/bin/oidldapd
008fe000-0090a000 rw-p 002fe000 fd:09 152688178
<FMWDIR>/bin/oidldapd
0090a000-0090d000 rw-p 00000000 00:00 0
0237a000-41d12000 rw-p 00000000 00:00 0
[heap]
2b8cfe5d4000-2b8d4a5d6000 rw-s 00000000 00:04 1030750211
/SYSV00007bd3 (deleted) <= NOTE
2b9e51d8c000-2b9e51dae000 r-xp 00000000 fd:00 37758220
/usr/lib64/ld-2.17.so
...
After one or more process failures, the OID instance may hang entirely. Below is an example stack from the dispatcher processes during an OID hang:
Thread 4 (Thread 0x2af311452700 (LWP 5635)):
#0 0x00000000005f083f in gslarddDispatcher ()
#1 0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#2 0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2af311c56700 (LWP 5748)):
#0 0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4 0x00000000025bab3f in sltsmna ()
#5 0x0000000000568b5d in gslupmaMutexAcquire ()
#6 0x00000000005f3ab9 in gslardlGetServer ()
#7 0x00000000005f3845 in gslardlAcceptConnAndSend ()
#8 0x00000000005f35f9 in gslardlAcceptConns ()
#9 0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2af312c5e700 (LWP 5756)):
#0 0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4 0x00000000025bab3f in sltsmna ()
#5 0x0000000000568b5d in gslupmaMutexAcquire ()
#6 0x00000000005f3ab9 in gslardlGetServer ()
#7 0x00000000005f3845 in gslardlAcceptConnAndSend ()
#8 0x00000000005f35f9 in gslardlAcceptConns ()
#9 0x00002af30785add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00002af30bea5ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2af306e7cf80 (LWP 4433)):
#0 0x00002af3078614ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002af30785cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2 0x00002af30785cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000025ba913 in ss_osw_wpthread_mutex_lock ()
#4 0x00000000025bab3f in sltsmna ()
#5 0x0000000000568b5d in gslupmaMutexAcquire ()
#6 0x000000000062a820 in gslardcController ()
#7 0x00000000005ea8c5 in gslarmcController ()
#8 0x000000000054cdf9 in gslmdispMain ()
#9 0x000000000054bf76 in main ()
...
May also see /var/log/messages such as below while grepping for kill:
Apr 17 07:32:32 <hostname> abrt-hook-ccpp: Process 7566 (frmweb) of user 1151 killed by SIGSEGV - dumping core
Apr 17 08:03:55 <hostname> abrt-hook-ccpp: Process 26218 (oidldapd) of user 1165 killed by SIGSEGV - ignoring (repeated crash)
Apr 17 08:03:55 <hostname> abrt-hook-ccpp: Process 26308 (oidldapd) of user 1165 killed by SIGSEGV - dumping core
Apr 17 08:04:08 <hostname> abrt-hook-ccpp: Process 26243 (oidldapd) of user 1165 killed by SIGSEGV - ignoring (repeated crash)
Apr 17 08:32:04 <hostname> abrt-hook-ccpp: Process 6746 (oidldapd) of user 1165 killed by SIGABRT - dumping core
Apr 17 09:31:40 <hostname> abrt-hook-ccpp: Process 10584 (frmweb) of user 1151 killed by SIGSEGV - dumping core
Apr 17 10:00:20 <hostname> abrt-hook-ccpp: Process 21899 (oidldapd) of user 1165 killed by SIGSEGV - dumping core
...
The dates/times match the stack dumps leading to a respawn.
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |