Oracle Glassfish Application Server Instance Hung Due To File System Mount Point Failure (Doc ID 1276496.1)

Last updated on NOVEMBER 05, 2016

Applies to:

Oracle GlassFish Server - Version 8.1 and later
Generic UNIX
Generic Linux
***Checked for relevance on 01-Mar-2013***

Symptoms

The application server is not responding

A pstack of the application server instance, shown below, indicates that Java thread id 197 is waiting for a response from Java thread id 198 ( 0xC6 == 198 decimal).  Therefore, there is a need to investigate Java thread id 198.

----------------- lwp# 197 --------------------------------
fef4c6e0 lwp_suspend (c6)
fef46cf4 suspend_fork (20, 1, 1ccc, 5d8d0a00, fefb5800, 1c00) + 84
fef3a7a4 fork (1, 17b, 0, 0, fefb5860, 5d8d0200) + e8
fe6c8ee4 Java_java_lang_UNIXProcess_forkAndExec (ffffffff, 0, 24c3df8, fe6e4000, a, 2) + 6d0
f900bc20 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+-27088
f900bbc4 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
f90058b8 * java/lang/UNIXProcess.init([B[BI[BI[BZ)V+62 (line 53)
f9005764 * java/lang/ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;Z)Ljava/lang/Process;+182 (line 65)
f9005874 * java/lang/ProcessBuilder.start()Ljava/lang/Process;+112 (line 451)
f9005874 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16 (line 591)
f9000218 * StubRoutines (1)
fe8dd540 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, a305b8, 5772b698, 5772b508, 5772b69c, 0) + 5b8
fe90de50 __1cKReflectionGinvoke6FnTinstanceKlassHandle_nMmethodHandle_nGHandle_inOobjArrayHandle_nJBasicType_4ipnGThread__pnHoopDesc__ (7e01d3b0, c, a630c4, c, 5772b790, fed6c000) + 1520
fe9e9bf0 __1cKReflectionNinvoke_method6FpnHoopDesc_nGHandle_nOobjArrayHandle_pnGThread__2_ (a630b8, a630ac, a630b0, a305b8, a630b4, 0) + 258
fe9e961c JVM_InvokeMethod (a30674, a630b0, 5772b940, 5772b93c, fed869e4, a305b8) + 2dc
fe6c0390 Java_sun_reflect_NativeMethodAccessorImpl_invoke0 (a30674, 5772b8dc, 5772b944, 5772b940, 5772b93c, fed86020) + 10
f91f8c38 * *sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
f914c28c * *sun/reflect/NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+87 (line 39)
f92072e4 * *sun/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6 (line 25)
f9206f68 * java/lang/Runtime.getRuntime()Ljava/lang/Runtime;+0
f9005874 * org/apache/tools/ant/taskdefs/Execute$Java13CommandLauncher.exec(Lorg/apache/tools/ant/Project;[Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+59 (line 831)
f9005874 * org/apache/tools/ant/taskdefs/Execute.launch(Lorg/apache/tools/ant/Project;[Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;Z)Ljava/lang/Process;+71 (line 436)
f9005874 * org/apache/tools/ant/taskdefs/Execute.execute()I+67 (line 450)
f90058b8 * org/apache/tools/ant/taskdefs/compilers/DefaultCompilerAdapter.executeExternalCompile([Ljava/lang/String;IZ)I+346 (line 443)
f90058b8 * org/apache/tools/ant/taskdefs/compilers/JavacExternal.execute()Z+63 (line 43)
f9005c70 * org/apache/tools/ant/taskdefs/Javac.compile()V+154 (line 936)
f9005764 * org/apache/tools/ant/taskdefs/Javac.execute()V+124 (line 758)
f9005764 * org/apache/jasper/compiler/Compiler.generateClass([Ljava/lang/String;)V+734 (line 409)
f9005764 * org/apache/jasper/compiler/Compiler.compile(ZZ)V+30 (line 502)
f9c957fc * *org/apache/jasper/compiler/Compiler.compile(Z)V+3 (line 481)
f9c957fc * *org/apache/jasper/compiler/Compiler.compile()V+2 (line 469)
f9c957fc * *org/apache/jasper/JspCompilationContext.compile()V+26 (line 516)
f9c970dc * *org/apache/jasper/servlet/JspServletWrapper.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V+120 (line 307)
f9692300 * *org/apache/jasper/servlet/JspServlet.serviceJspFile(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;Ljav
a/lang/Throwable;Z)V+129 (line 301)
f932c24c * *org/apache/jasper/servlet/JspServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+440 (line 251)
f9122c80 * *javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30 (line 860)
f980c96c * *sun/reflect/GeneratedMethodAccessor82.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+48
f92072e4 * *sun/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6 (line 25)
f9206f68 * *java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+111 (line 585)
f9409f60 * *org/apache/catalina/security/SecurityUtil$1.run()Ljava/lang/Object;+12 (line 249)
f9000218 * StubRoutines (1)
fe8dd540 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, a305b8, 5772c794, 5772c698, 5772c798, 0) + 5b8
fe8eda10 JVM_DoPrivileged (a30674, 80000001, a630a8, 4, c, a630a4) + 560
fe6bee84 Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2 (a30674, 5772ca24, 5772ca8
8, 5772ca84, 1, f) + 14
f91aee60 * *java/security/AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;
f9d8b150 * *javax/security/auth/Subject.doAsPrivileged(Ljavax/security/auth/Subject;Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Lja
va/lang/Object;+60 (line 517)
f9430b9c * *org/apache/catalina/security/SecurityUtil.execute(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;Ljava/security/Principal;)V+136 (line 28
2)
f943237c * *org/apache/catalina/security/SecurityUtil.doAsPrivilege(Ljava/lang/String;Ljavax/servlet/Servlet;[Ljava/lang/Class;[Ljava/lang/Object;Ljava/security/Prin
cipal;)V+70 (line 165)
f93d702c * *org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+338 (line 257)
f96d4db8 * *org/apache/catalina/core/ApplicationFilterChain.access$000(Lorg/apache/catalina/core/ApplicationFilterChain;Ljavax/servlet/ServletRequest;Ljavax/servlet/
ServletResponse;)V+3 (line 55)
f96d4db8 * *org/apache/catalina/core/ApplicationFilterChain$1.run()Ljava/lang/Object;+12 (line 161)


In the stack trace, you can see that Java thread id 198 is trying to resolve a file system path:

----------------- lwp# 198 --------------------------------
00000000 resolvepath (25e5ac0, 576dcd60, 400)
----------------- lwp# 199 --------------------------------
00000000 resolvepath (2631760, 5768ce30, 400)
----------------- lwp# 200 --------------------------------
00000000 resolvepath (29e8230, 5763cdd8, 400)


Checking with truss reveals that the thread id 198 is waiting for a response from the file system:

28744/198: resolvepath("/opt/XXX/21122010/3208/XYZ/00000000245074/3385470/XYZ.pdf ", 0x576DCD60, 1024) (sleeping...)



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms