JVM Crash Due to Out Of Memory Condition on a ZFS File System (Doc ID 1240333.1)

Last updated on APRIL 09, 2017

Applies to:

Java Platform, Standard Edition - Version 1.4.2 and later
Oracle Solaris on x86 (32-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)
This was a crash of the JVM. It produced an application core file plus a HotSpot error log.
This type of crash can come from any standalone java application.
***Checked for relevance on 08-JUL-2012***
***Checked for relevance on 29-DEC-2013***

Symptoms

A Java process crashes due to an out of memory (OOM) condition.

Here is an example of the output from pstack running on the core file from the crash:

--- called from signal handler with signal 11 (SIGSEGV) ---
 fffffd7fff2d720a setup_context () + 7a
 fffffd7fff2d36b5 _thrp_create () + 255
 fffffd7fff2d3a0f thr_create () + 3f
 fffffd7ffeaee1b7 int os::create_thread(Thread*,os::ThreadType,unsigned long)() + 277
 fffffd7ffeaf5d64 JavaThread::JavaThread #Nvariant 1(void(*)(JavaThread*,Thread*),unsigned long) () + 94
 fffffd7ffeafbb06 JVM_StartThread () + 166
 fffffd7ff720c722 * java/lang/Thread.start0()V+0
 fffffd7ff7202efe * java/lang/Thread.start()V+29 (line 574)


Here is the associated stack trace from the Hot Spot error log file:

# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xfffffd7fff2d720a, pid=14388, tid=282 # # JavaVM: Java HotSpot(TM)
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xfffffd7fbf9ffff8
# Problematic frame:
# C  [libc.so.1+0xd720a]  _thr_slot_offset+0x25a #


Stack: [0xfffffd7379200000,0xfffffd7379400000),  sp=0xfffffd73793fec30, free space=2043k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.1+0xd720a] _thr_slot_offset+0x25a do_decomp+0x56a
C  [libc.so.1+0xd36b5] _pthread_cond_destroy+0x16b5 mark_dead_and_buried+0x25
C  [libc.so.1+0xd3a0f]  thr_create+0x3f thr_setprio+0x2f
V  [libjvm.so+0x4ee1b7]
V  [libjvm.so+0x4f5d64]
V  [libjvm.so+0x4fbb06]
j  java.lang.Thread.start0()V+0


This stack trace reveals that the crash was because the system was either short of or out of swap space.  In this particular case, the JVM is trying to create a new Java thread and there was not enough memory to create the stack for the Java thread.

Changes

The most important contributing factor to this problem is the use of ZFS, which has a different file system caching strategy and mechanism to UFS filesystems.

This problem can difficult to diagnose because it depends on the memory demands being placed on the system.  For example, if a system is running multiple Solaris zones which are using ZFS filesystems, and the zones are sharing the underlying resources of the machine they are running on, then how much memory is needed will depend on the memory demands of each zone. 

If additional JVMs are run, for example additional Application Servers, then that zone may use more memory thereby reducing the amount available to other zones on the system.  This could lead to this sort of JVM crash on a zone that was hitherto considered perfectly stable.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms