My Oracle Support Banner

JVM Crash Due to Out Of Memory Condition on a ZFS File System (Doc ID 1240333.1)

Last updated on JULY 10, 2018

Applies to:

Java Platform, Standard Edition - Version 1.4.2 and later
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on x86 (32-bit)
This is a crash of the JVM that produces an application core file plus a HotSpot fatal error log, hs_err_pid.log (hs_err log).
This type of crash can come from any standalone java application.
***Checked for relevance on 08-JUL-2012***
***Checked for relevance on 29-DEC-2013***

Symptoms

A Java process crashes due to an out of memory (OOM) condition.  The crash occurs on a zoned Solaris system which uses a ZFS type filesystem.  It occurs only on one system, even though the application runs on another system that is not affected1


Here is an example of the output from pstack running on the core file from the crash:

--- called from signal handler with signal 11 (SIGSEGV) ---
 fffffd7fff2d720a setup_context () + 7a
 fffffd7fff2d36b5 _thrp_create () + 255
 fffffd7fff2d3a0f thr_create () + 3f
 fffffd7ffeaee1b7 int os::create_thread(Thread*,os::ThreadType,unsigned long)() + 277
 fffffd7ffeaf5d64 JavaThread::JavaThread #Nvariant 1(void(*)(JavaThread*,Thread*),unsigned long) () + 94
 fffffd7ffeafbb06 JVM_StartThread () + 166
 fffffd7ff720c722 * java/lang/Thread.start0()V+0
 fffffd7ff7202efe * java/lang/Thread.start()V+29 (line 574)


Here is the associated stack trace from the hs_err log:

# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xfffffd7fff2d720a, pid=14388, tid=282 # # JavaVM: Java HotSpot(TM)
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xfffffd7fbf9ffff8
# Problematic frame:
# C  [libc.so.1+0xd720a]  _thr_slot_offset+0x25a #


Stack: [0xfffffd7379200000,0xfffffd7379400000),  sp=0xfffffd73793fec30, free space=2043k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.1+0xd720a] _thr_slot_offset+0x25a do_decomp+0x56a
C  [libc.so.1+0xd36b5] _pthread_cond_destroy+0x16b5 mark_dead_and_buried+0x25
C  [libc.so.1+0xd3a0f]  thr_create+0x3f thr_setprio+0x2f
V  [libjvm.so+0x4ee1b7]
V  [libjvm.so+0x4f5d64]
V  [libjvm.so+0x4fbb06]
j  java.lang.Thread.start0()V+0


This stack trace reveals that the crash occurs when the the system has too little swap space.  The JVM is trying to create a new Java thread and there is not enough native memory to create the stack for the Java thread.

Changes

The most important contributing factor to this problem is the use of ZFS type filesystem, which has a different filesystem caching strategy and mechanism compared to the traditional UFS filesystems.
Changes that cause additional memory usage, such as running additional processes, in any of the Solaris zones can trigger the issue.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.