JVM Crash Due to Out Of Memory Condition on a ZFS File System
Last updated on APRIL 09, 2017
Applies to:Java Platform, Standard Edition - Version 1.4.2 and later
Oracle Solaris on x86 (32-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)
This was a crash of the JVM. It produced an application core file plus a HotSpot error log.
This type of crash can come from any standalone java application.
***Checked for relevance on 08-JUL-2012***
***Checked for relevance on 29-DEC-2013***
A Java process crashes due to an out of memory (OOM) condition.
Here is an example of the output from pstack running on the core file from the crash:
fffffd7fff2d720a setup_context () + 7a
fffffd7fff2d36b5 _thrp_create () + 255
fffffd7fff2d3a0f thr_create () + 3f
fffffd7ffeaee1b7 int os::create_thread(Thread*,os::ThreadType,unsigned long)() + 277
fffffd7ffeaf5d64 JavaThread::JavaThread #Nvariant 1(void(*)(JavaThread*,Thread*),unsigned long) () + 94
fffffd7ffeafbb06 JVM_StartThread () + 166
fffffd7ff720c722 * java/lang/Thread.start0()V+0
fffffd7ff7202efe * java/lang/Thread.start()V+29 (line 574)
Here is the associated stack trace from the Hot Spot error log file:
# SIGSEGV (0xb) at pc=0xfffffd7fff2d720a, pid=14388, tid=282 # # JavaVM: Java HotSpot(TM)
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xfffffd7fbf9ffff8
# Problematic frame:
# C [libc.so.1+0xd720a] _thr_slot_offset+0x25a #
Stack: [0xfffffd7379200000,0xfffffd7379400000), sp=0xfffffd73793fec30, free space=2043k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0xd720a] _thr_slot_offset+0x25a do_decomp+0x56a
C [libc.so.1+0xd36b5] _pthread_cond_destroy+0x16b5 mark_dead_and_buried+0x25
C [libc.so.1+0xd3a0f] thr_create+0x3f thr_setprio+0x2f
This stack trace reveals that the crash was because the system was either short of or out of swap space. In this particular case, the JVM is trying to create a new Java thread and there was not enough memory to create the stack for the Java thread.
The most important contributing factor to this problem is the use of ZFS, which has a different file system caching strategy and mechanism to UFS filesystems.
This problem can difficult to diagnose because it depends on the memory demands being placed on the system. For example, if a system is running multiple Solaris zones which are using ZFS filesystems, and the zones are sharing the underlying resources of the machine they are running on, then how much memory is needed will depend on the memory demands of each zone.
If additional JVMs are run, for example additional Application Servers, then that zone may use more memory thereby reducing the amount available to other zones on the system. This could lead to this sort of JVM crash on a zone that was hitherto considered perfectly stable.
Sign In with your My Oracle Support account
Don't have a My Oracle Support account? Click to get started
My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms