JVM hang in Runtime.exec, block In malloc_atfork, free_atfork (Doc ID 1338607.1)

Last updated on SEPTEMBER 23, 2011

Applies to:

Java Platform, Standard Edition - Version: 1.6.0 and later   [Release: 1.6 and later ]
Linux x86-64

Symptoms

The symptom was reported on RHEL 5.3 with Java 6u14.  The symptom can happen on Java 6 up to update 22.

A Java program which performs Runtime.exec on multiple threads occasionally hangs up.  In the native stack trace, we see Runtime.exec is blocked around fork() in glibc.

As seen in this example stack trace, the thread is blocked in Runtime.exec() by glibc list_all_lock:

#0 0x00000032ee0dee6e in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x00000032ee06ebe1 in _L_lock_2271 () from /lib64/libc.so.6
#2 0x00000032ee06dfdf in __GI__IO_list_lock () from /lib64/libc.so.6 _IO_list_lock ();
#3 0x00000032ee09980f in fork () from /lib64/libc.so.6
#4 0x00002aaaaabe5eee in Java_java_lang_UNIXProcess_forkAndExec ()
from /usr/local/jdk1.6.0_14/jre/lib/amd64/libjava.so
#5 0x00002aaaab146ece in ?? ()
#6 0x0000000000000000 in ?? ()

This method, list_all_lock, in glibc coordinates fork and stdio:

(gdb) p list_all_lock // if glibc debug info is available
$1 = {lock = 2, cnt = -1, owner = 0x40e8f940}
(gdb) x/3wx &list_all_lock // if glibc debug info is not available
0x3d45752940 <list_all_lock>:    0x00000002    0xffffffff    0x40eea940

cnt is wrong.  It should be 0 (unlocked) or larger (locked).

Malloc and free are often blocked in malloc_atfork/free_atfork when the symptom was seen.  When fork is active, malloc/free would take *_atfork path and block until fork completes.

Thread 107 (process 12285):
#0 0x00000032ee0dee6e in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x00000032ee07679a in _L_lock_13544 () from /lib64/libc.so.6
#2 0x00000032ee074df6 in free_atfork () from /lib64/libc.so.6
#3 0x00000032ee0758a7 in free () from /lib64/libc.so.6
#4 0x00002aaaaabe601f in Java_java_lang_UNIXProcess_forkAndExec ()
from /usr/local/jdk1.6.0_14/jre/lib/amd64/libjava.so
#5 0x00002aaaab146ece in ?? ()
#6 0x00002b1d00000000 in ?? ()
#7 0x0000000000000000 in ?? ()

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms