My Oracle Support Banner

EM13c: OMS Crashes After Startup with java.lang.OutOfMemoryError: unable to create new native thread error (Doc ID 2484900.1)

Last updated on APRIL 19, 2023

Applies to:

Enterprise Manager Base Platform - Version 13.2.2.0.0 and later
Information in this document applies to any platform.

Symptoms

13cR2 OMS crashes after startup with out of memory error.

<gc_inst>/em/EMGC_OMSn/sysman/log/emctl.msg

HealthMonitor Dec 14, 2018 4:58:25 AM JobDispatcher error:
Restarting OMS due to out of memory condition.
Enabling java heap dumps will help Oracle Support diagnose
this further. Java heap dumps can be enabled by adding
-XX:+HeapDumpOnOutOfMemoryError to JVM startup options
via the WLS administration server console or in <gc_inst>/user_projects/domains/GCDomain/bin/startEMServer.[sh|cmd] file.
Heap dumps are generated by default in <gc_inst>/user_projects/domains/GCDomain,
and an alternate location can be specified by using the
-XX:HeapDumpPath=<path> JVM startup option.
Please consult My Oracle Support Note 794165.1 for detailed instructions.


<gc_inst>/em/EMGC_OMSn/sysman/log/oms_diag_info_<Timestamp>.msg

HealthMonitor Dec 14, 2018 4:58:25 AM JobDispatcher error:
Restarting OMS due to out of memory condition.
Enabling java heap dumps will help Oracle Support diagnose
this further. Java heap dumps can be enabled by adding
-XX:+HeapDumpOnOutOfMemoryError to JVM startup options
via the WLS administration server console or in <gc_inst>/user_projects/domains/GCDomain/bin/startEMServer.[sh|cmd] file.
Heap dumps are generated by default in <gc_inst>/user_projects/domains/GCDomain,
and an alternate location can be specified by using the
-XX:HeapDumpPath=<path> JVM startup option.
Please consult My Oracle Support Note 794165.1 for detailed instructions.
...
Profiler JobDispatcher Not Used.Didn't Collected Any Data



Profiler PbsAdminMsgListener Collected, Profiled Threads Data
Thread | Iteration | Method | Start Time | End Time | Elapsed Time in milli sec

PbsAdminMsgListener-100 | 23 | healthMonitorPing | 2018-12-14 04:57:24 | 2018-12-14 04:57:24 | 0 ms
PbsAdminMsgListener-100 | 23 | getAdminMsg | 2018-12-14 04:57:24 | 2018-12-14 04:57:41 | 16526 ms
PbsAdminMsgListener-100 | 23 | getConnection | 2018-12-14 04:57:24 | 2018-12-14 04:57:24 | 0 ms
PbsAdminMsgListener-100 | 23 | processAdminMessage | 2018-12-14 04:57:41 | 2018-12-14 04:57:41 | 0 ms
PbsAdminMsgListener-100 | 24 | healthMonitorPing | 2018-12-14 04:57:41 | 2018-12-14 04:57:41 | 0 ms
PbsAdminMsgListener-100 | 24 | getAdminMsg | 2018-12-14 04:57:41 | 2018-12-14 04:58:01 | 20014 ms
PbsAdminMsgListener-100 | 24 | getConnection | 2018-12-14 04:57:41 | 2018-12-14 04:57:41 | 0 ms
PbsAdminMsgListener-100 | 25 | healthMonitorPing | 2018-12-14 04:58:01 | 2018-12-14 04:58:01 | 0 ms
PbsAdminMsgListener-100 | 25 | getAdminMsg | 2018-12-14 04:58:01 | 2018-12-14 04:58:21 | 20014 ms
PbsAdminMsgListener-100 | 25 | getConnection | 2018-12-14 04:58:01 | 2018-12-14 04:58:01 | 0 ms
PbsAdminMsgListener-100 | 26 | healthMonitorPing | 2018-12-14 04:58:21 | 2018-12-14 04:58:21 | 0 ms
PbsAdminMsgListener-100 | 26 | getAdminMsg | 2018-12-14 04:58:21 | 2018-12-14 04:58:41 | 20022 ms
PbsAdminMsgListener-100 | 26 | getConnection | 2018-12-14 04:58:21 | 2018-12-14 04:58:21 | 0 ms
PbsAdminMsgListener-100 | 27 | healthMonitorPing | 2018-12-14 04:58:41 | 2018-12-14 04:58:41 | 0 ms
PbsAdminMsgListener-100 | 27 | getAdminMsg | 2018-12-14 04:58:41 |  -  |  Method running for 8135 ms
PbsAdminMsgListener-100 | 27 | getConnection | 2018-12-14 04:58:41 | 2018-12-14 04:58:41 | 0 ms


<gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMSn/logs/EMGC_OMSn.log

<Dec 14, 2018 3:09:43 AM CET> <Warning> <HTTP> <OMSHOSTNAME> <EMGC_OMS1> <[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <4ca2340d-9c93-4743-89e0-4514a28157f0-00000002> <1544753383771> <BEA-101162> <User defined listener oracle.sysman.eml.app.ContextInitializer failed: java.lang.OutOfMemoryError: unable to create new native thread.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at oracle.jdbc.pool.OracleConnectionCacheManager.checkAndStartThread(OracleConnectionCacheManager.java:1492)
at oracle.jdbc.pool.OracleImplicitConnectionCache.setConnectionCacheProperties(OracleImplicitConnectionCache.java:2264)
at oracle.jdbc.pool.OracleImplicitConnectionCache.<init>(OracleImplicitConnectionCache.java:242)

 

Verifying the Heap Memory as per the EM 13c, 12c: Troubleshooting Steps for Resolving java.lang.OutOfMemoryError Which is Causing the OMS to Hang / Restart (Doc ID 2052851.1)
indicates that Heap max is already high (in this case 8GB).

$emctl get property -name OMS_HEAP_MAX
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SYSMAN password:
Value for property OMS_HEAP_MAX for oms local_oms is 8192M

 

There is no shortage of available free memory on the system:

free -m
            total       used       free     shared    buffers     cached
Mem:         32236      22228      10007          0        965      18521
-/+ buffers/cache:       2741      29494
Swap:         1023          0       1023

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.