My Oracle Support Banner

OVD and OID 11g Returning Native Thread Error Continuously (ldap_search: Operations error | ldap_search: additional info: unable to create new native thread) | java.lang.OutOfMemoryError: unable to create new native thread (Doc ID 1926751.1)

Last updated on JUNE 06, 2024

Applies to:

Oracle Internet Directory - Version 11.1.1 to 11.1.1.7.0 [Release 11g]
Oracle Virtual Directory - Version 11.1.1.0 to 11.1.1.7.0
Information in this document applies to any platform.

Symptoms

Oracle Virtual Directory (OVD) and Oracle Internet Directory (OID) 11g installed on same server.

OVD-OID servers giving native thread error continuously.

The server is periodically throwing the errors below:

bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
ldap_search: Operations error
ldap_search: additional info: unable to create new native thread

Once getting this error, all services on the box are hung.   Cannot execute any command on the server, and cannot perform any action or authentication from any application such as Oracle Directory Services Manager (ODSM) or Oracle Access Manager (OAM) to OVD.

OVD and WebLogic (where ODSM is deployed) tuning has already been applied.

Oracle Linux Support analyzed the issue and it is related to the number of open files, i.e., more than the hard limit given to Unix user.  However this unix setting has been increased and still the issue persists.  The logs are as follows:

root@myserver ~]# tail -20 /var/log/messages
Sep 9 13:48:28 myserver abrtd: Executable '/u01/app/product/Middleware/Oracle_IDM1/bin/emagent' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 9 13:48:28 myserver abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-09-09-13:48:26-9897' exited with 1
Sep 9 13:48:28 myserver abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-09-09-13:48:26-9897'
Sep 9 14:18:55 myserver kernel: gdb[14427]: segfault at 0 ip 0000003645e6e820 sp 00007fff24f410d8 error 4 in libc-2.12.so[3645e00000+18b000]
Sep 9 14:18:55 myserver abrt[14429]: Can't open '/proc/14427/maps': No such file or directory
Sep 9 14:18:55 myserver abrt[14429]: Can't open '/proc/14427/limits': No such file or directory
Sep 9 14:18:55 myserver abrt[14429]: Can't open '/proc/14427/cgroup': No such file or directory
Sep 9 14:18:55 myserver abrt[14429]: Saved core dump of pid 14427 (/usr/bin/gdb) to /var/spool/abrt/ccpp-2014-09-09-14:18:55-14427 (65536 bytes)
Sep 9 14:18:55 myserver abrtd: Directory 'ccpp-2014-09-09-14:18:55-14427' creation detected
Sep 9 14:19:02 myserver abrt[14567]: Saved core dump of pid 11191 (/u01/app/product/Middleware/Oracle_IDM1/bin/emagent) to /var/spool/abrt/ccpp-2014-09-09-14:19:00-11191 (300744704 bytes)
Sep 9 14:19:02 myserver abrtd: Directory 'ccpp-2014-09-09-14:19:00-11191' creation detected
Sep 9 14:19:39 myserver abrtd: Sending an email...
Sep 9 14:19:39 myserver abrtd: Email was sent to: root@localhost
Sep 9 14:19:39 myserver abrtd: Can't open 'maps': [Errno 2] No such file or directory: 'maps'
Sep 9 14:19:39 myserver abrtd: Error writing to 'dso_list'
Sep 9 14:19:39 myserver abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-09-09-14:18:55-14427' exited with 1
Sep 9 14:19:39 myserver abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-09-09-14:18:55-14427'
Sep 9 14:19:39 myserver abrtd: Executable '/u01/app/product/Middleware/Oracle_IDM1/bin/emagent' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 9 14:19:39 myserver abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-09-09-14:19:00-11191' exited with 1
Sep 9 14:19:39 myserver abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-09-09-14:19:00-11191'


There is not one command or search which hangs the server, it is always something different, and even if browsing ODSM console.

For example, when there is no load on server and no user using the system, seeing out of memory/resource not available errors:

####<Sep 7, 2014 8:56:42 AM CEST> <Info> <Health> <HOSTNAME> <wls_ods1> <weblogic.GCMonitor> <> <> <5a9a85b473496ef9:-4232bf45:14849d06ff5:-8000-0000000000000073> <1410073002441> <BEA-310002> <27% of the total memory in the server is free>
####<Sep 7, 2014 8:57:29 AM CEST> <Info> <EJB> <HOSTNAME> <wls_ods1> <mprovprofile_I> <> <> <0000KX9o8bF3NAyLG6G7yf1K2fHL000002> <1410073049084> <BEA-010227> <EJB Exception occurred during invocation from home or business: oracle.ldap.odip.web.DIPProv_o70i40_HomeImpl@847ea04 threw exception: java.lang.OutOfMemoryError: unable to create new native thread>
####<Sep 7, 2014 8:57:42 AM CEST> <Info> <Health> <HOSTNAME> <wls_ods1> <weblogic.GCMonitor> <> <> <5a9a85b473496ef9:-4232bf45:14849d06ff5:-8000-0000000000000073> <1410073062443> <BEA-310002> <3% of the total memory in the server is free>
####<Sep 7, 2014 8:58:19 AM CEST> <Info> <EJB> <HOSTNAME> <wls_ods1> <mprovprofile_E> <> <> <0000KX9o8bF3NAyLG6G7yf1K2fHL000002> <1410073099732> <BEA-010227> <EJB Exception occurred during invocation from home or business: oracle.ldap.odip.web.DIPProv_o70i40_HomeImpl@847ea04 threw exception: java.lang.OutOfMemoryError: unable to create new native thread>
####<Sep 7, 2014 8:58:19 AM CEST> <Info> <Common> <HOSTNAME> <wls_ods1> <mprovprofile_E> <> <> <0000KX9o8bF3NAyLG6G7yf1K2fHL000002> <1410073099801> <BEA-000628> <Created "1" resources for pool "schedulerDS", out of which "1" are available and "0" are unavailable.>
####<Sep 7, 2014 8:58:29 AM CEST> <Info> <EJB> <HOSTNAME> <wls_ods1> <mprovprofile_I> <> <> <0000KX9o8bF3NAyLG6G7yf1K2fHL000002> <1410073109106> <BEA-010227> <EJB Exception occurred during invocation from home or business: oracle.ldap.odip.web.DIPProv_o70i40_HomeImpl@847ea04 threw exception: java.lang.OutOfMemoryError: unable to create new native thread>
####<Sep 7, 2014 8:58:42 AM CEST> <Info> <Health> <HOSTNAME> <wls_ods1> <weblogic.GCMonitor> <> <> <5a9a85b473496ef9:-4232bf45:14849d06ff5:-8000-0000000000000073> <1410073122445> <BEA-310002> <16% of the total memory in the server is free>
####<Sep 7, 2014 8:59:19 AM CEST> <Info> <EJB> <HOSTNAME> <wls_ods1> <mprovprofile_E> <> <> <0000KX9o8bF3NAyLG6G7yf1K2fHL000002> <1410073159731> <BEA-010227> <EJB Exception occurred during invocation from home or business: oracle.ldap.odip.web.DIPProv_o70i40_HomeImpl@847ea04 threw exception: java.lang.OutOfMemoryError: unable to create new native thread>

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.