JVM Crash Due to Out Of Memory Condition on a ZFS File System
(Doc ID 1240333.1)
Last updated on JULY 10, 2018
Applies to:Java Platform, Standard Edition - Version 1.4.2 and later
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on x86 (32-bit)
This is a crash of the JVM that produces an application core file plus a HotSpot fatal error log, hs_err_pid
This type of crash can come from any standalone java application.
***Checked for relevance on 08-JUL-2012***
***Checked for relevance on 29-DEC-2013***
A Java process crashes due to an out of memory (OOM) condition. The crash occurs on a zoned Solaris system which uses a ZFS type filesystem. It occurs only on one system, even though the application runs on another system that is not affected1.
Here is an example of the output from pstack running on the core file from the crash:
fffffd7fff2d720a setup_context () + 7a
fffffd7fff2d36b5 _thrp_create () + 255
fffffd7fff2d3a0f thr_create () + 3f
fffffd7ffeaee1b7 int os::create_thread(Thread*,os::ThreadType,unsigned long)() + 277
fffffd7ffeaf5d64 JavaThread::JavaThread #Nvariant 1(void(*)(JavaThread*,Thread*),unsigned long) () + 94
fffffd7ffeafbb06 JVM_StartThread () + 166
fffffd7ff720c722 * java/lang/Thread.start0()V+0
fffffd7ff7202efe * java/lang/Thread.start()V+29 (line 574)
Here is the associated stack trace from the hs_err log:
# SIGSEGV (0xb) at pc=0xfffffd7fff2d720a, pid=14388, tid=282 # # JavaVM: Java HotSpot(TM)
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xfffffd7fbf9ffff8
# Problematic frame:
# C [libc.so.1+0xd720a] _thr_slot_offset+0x25a #
Stack: [0xfffffd7379200000,0xfffffd7379400000), sp=0xfffffd73793fec30, free space=2043k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0xd720a] _thr_slot_offset+0x25a do_decomp+0x56a
C [libc.so.1+0xd36b5] _pthread_cond_destroy+0x16b5 mark_dead_and_buried+0x25
C [libc.so.1+0xd3a0f] thr_create+0x3f thr_setprio+0x2f
This stack trace reveals that the crash occurs when the the system has too little swap space. The JVM is trying to create a new Java thread and there is not enough native memory to create the stack for the Java thread.
The most important contributing factor to this problem is the use of ZFS type filesystem, which has a different filesystem caching strategy and mechanism compared to the traditional UFS filesystems.
Changes that cause additional memory usage, such as running additional processes, in any of the Solaris zones can trigger the issue.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!