DSEE Directory Server Stops Responding to Connections and in Some Cases Can Recover After a Few Minutes (Doc ID 1388346.1)

Last updated on OCTOBER 11, 2016

Applies to:

Oracle Directory Server Enterprise Edition - Version 5.1 to 11.1.1.5.0 [Release 5.0 to 11gR1]
Oracle Directory Server Enterprise Edition - Version 5.2 and later
Information in this document applies to any platform.
***Checked for relevance on 15-Jul-2013***

Symptoms

Symptoms can include one or more of the following:


- The Directory Server goes inactive or seems to hang.

- The Directory Server goes inactive for some time [minutes, or longer] but recovers on its own.

- In some cases, the Directory Server cannot be stopped and has to be killed.

 

 

Changes

The cause is due to an excess buildup of Bad/Stale SSL connections that can lock the Directory Servers threads.    In some cases, it can clear up on its own.

To determine if you are hitting this issue you can use the following methods:

        1.) If a DirTracer is taken WHILE the issue is happening you can use the cn=monitor searches from the DirTracer capture.   For example, in one case we could see there are 176 LDAPS (SSL) connections in the       "cn=monitor" output:

[/<DirTracer/capture>/ds-srch-monitor]#grep -c LDAPS *
monitor.111215-115714:176
monitor.111215-115721:176
monitor.111215-115726:176
monitor.111215-115732:176
monitor.111215-115738:176

        2.) In our example, DirTracer also took a series of pstacks.   We can see a portion of the ssl Hand Shakes is happening.  Reference the ssl functions in the pstack summaries example below.

   1 ./ns-slapd
   1 _lwp_start _pt_root ???????? ldap_frontend_main_using_core_api ber_get_next get_tag BerRead ber_filbuf ???????? ssl_Recv ssl_SecureRecv ssl_Do1stHandshake ssl_GatherRecord1stHandshake ssl3_GatherCompleteHandshake ssl3_GatherData ssl_DefRecv pt_Recv pt_poll_now poll pollsys
   1 _lwp_start _pt_root referint_thread_func ???????? PR_WaitCondVar pt_TimedWait pthread_cond_timedwait cond_timedwait _cond_timedwait cond_wait_common cond_wait_queue lwp_park
   1 _lwp_start _pt_root time_thread DS_Sleep select pselect pollsys
   1 _start main slapd_daemon ???????? PR_Unlock mutex_unlock lwp_yield
   2 _lwp_start _pt_root ???????? ???????? ldap_result nsldapi_result_nolock wait4msg nsldapi_iostatus_poll prldap_poll _pr_poll_with_poll poll pollsys
   2 _lwp_start _pt_root ???????? ldap_frontend_main_using_core_api ber_get_next get_tag BerRead ber_filbuf ???????? ssl_Recv DoRecv ssl3_GatherAppDataRecord ssl3_GatherCompleteHandshake ssl3_GatherData ssl_DefRecv pt_Recv pt_poll_now poll pollsys
   2 _lwp_start _pt_root ???????? repl5_trs_inc_run protocol_sleep_mask_events PR_WaitCondVar pt_TimedWait pthread_cond_timedwait cond_timedwait _cond_timedwait cond_wait_common cond_wait_queue lwp_park
   3 _lwp_start _pt_root ???????? PR_WaitCondVar pt_TimedWait pthread_cond_timedwait cond_timedwait _cond_timedwait cond_wait_common cond_wait_queue lwp_park
   3 _lwp_start _pt_root ???????? repl5_trs_inc_run protocol_sleep_mask_events PR_WaitCondVar pthread_cond_wait cond_wait cond_wait_queue lwp_park
   3 _lwp_start _pt_root ???????? slapi_wait_condvar PR_WaitCondVar pthread_cond_wait cond_wait cond_wait_queue lwp_park
   5 _lwp_start _pt_root ???????? DS_Sleep select pselect pollsys
  27 _lwp_start _pt_root ???????? ???????? PR_WaitCondVar pt_TimedWait pthread_cond_timedwait cond_timedwait _cond_timedwait cond_wait_common cond_wait_queue lwp_park
  51



Using a series of pstacks, you can visually scan the threads for all functions that begin with the "ssl" or "ssl3."  You can also use the attached Pstacker script to create pstack summaries of the captured pstacks. 

Pstacker, extracts the functions from pstack threads and group the like threads functions together.  Each line contains the number of like threads the pstack had.  The final line shows the total number of threads the process has.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms