Server.log File Removal May Cause A Hang If Log File Rotation is On In GlassFish Server 3.1

(Doc ID 1339977.1)

Last updated on NOVEMBER 05, 2016

Applies to:

Oracle GlassFish Server - Version 3.1 to 3.1 [Release 3.1]
Information in this document applies to any platform.

Symptoms

A running Oracle GlassFish Server 3.1 process stops responding after it's server.log file is removed.
Using the Java jstack on the GlassFish process to trace the issue, the process is hung in logging code:

"http-thread-pool-8080(3)" daemon prio=3 tid=0x084dd800 nid=0x34 waiting on condition [0xcdb57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdb88d2b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at com.sun.enterprise.server.logging.GFFileHandler.publish(GFFileHandler.java:559)
at java.util.logging.Logger.log(Logger.java:458)
at com.sun.logging.LogDomains$1.log(LogDomains.java:354)
at java.util.logging.Logger.doLog(Logger.java:480)
at java.util.logging.Logger.logp(Logger.java:596)
at com.sun.common.util.logging.LoggingOutputStream.flush(LoggingOutputStream.java:104)
at java.io.PrintStream.write(PrintStream.java:432)
- locked <0xdb87eff8> (a com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream)
at com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream.write(LoggingOutputStream.java:302)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
- locked <0xdb89cd08> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
at java.io.PrintStream.write(PrintStream.java:477)
- locked <0xdb87eff8> (a com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream)
at java.io.PrintStream.print(PrintStream.java:619)
at com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream.print(LoggingOutputStream.java:207)
at org.apache.felix.gogo.runtime.threadio.ThreadPrintStream.print(ThreadPrintStream.java:150)
at org.apache.jsp.t100k_jsp._jspService(t100k_jsp.java from :46)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1534)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at com.sun.enterprise.v3.services.impl.SnifferAdapter.service(SnifferAdapter.java:178)
- locked <0xdbdce770> (a org.glassfish.internal.data.ContainerRegistry)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)


Under a normal operation,  the drainer thread (GFFileHandler) that process the logging. When the server.log is removed, it is seen that this thread also disappears. The drainer thread typically looks like this :

"Thread-2" prio=3 tid=0x09137400 nid=0x10 waiting on condition [0xce76c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdba063f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:514)
at com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:166)

 

Note: Stoping the GlassFish process using "asadmin stop-domain" will stall and time out. Hence to recover or to stop the process, the process needs to be killed by a process kill.



Changes

This symptom can happen only if the GlassFish server.log file is removed while GlassFish Server 3.1 is in started up and there is a thread writing extensively to the server.log file.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms