My Oracle Support Banner

Server.log File Removal May Cause A Hang If Log File Rotation is On In GlassFish Server 3.1 (Doc ID 1339977.1)

Last updated on JANUARY 31, 2019

Applies to:

Oracle GlassFish Server - Version 3.1 to 3.1 [Release 3.1]
Information in this document applies to any platform.

Symptoms

A running Oracle GlassFish Server 3.1 process stops responding after it's server.log file is removed.
Using the Java jstack on the GlassFish process to trace the issue, the process is hung in logging code:

"http-thread-pool-8080(3)" daemon prio=3 tid=0x084dd800 nid=0x34 waiting on condition [0xcdb57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdb88d2b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at com.sun.enterprise.server.logging.GFFileHandler.publish(GFFileHandler.java:559)
at java.util.logging.Logger.log(Logger.java:458)
at com.sun.logging.LogDomains$1.log(LogDomains.java:354)
at java.util.logging.Logger.doLog(Logger.java:480)
at java.util.logging.Logger.logp(Logger.java:596)
at com.sun.common.util.logging.LoggingOutputStream.flush(LoggingOutputStream.java:104)
at java.io.PrintStream.write(PrintStream.java:432)
- locked <0xdb87eff8> (a com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream)
at com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream.write(LoggingOutputStream.java:302)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
- locked <0xdb89cd08> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
at java.io.PrintStream.write(PrintStream.java:477)
- locked <0xdb87eff8> (a com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream)
at java.io.PrintStream.print(PrintStream.java:619)
at com.sun.common.util.logging.LoggingOutputStream$LoggingPrintStream.print(LoggingOutputStream.java:207)
at org.apache.felix.gogo.runtime.threadio.ThreadPrintStream.print(ThreadPrintStream.java:150)
at org.apache.jsp.t100k_jsp._jspService(t100k_jsp.java from :46)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1534)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at com.sun.enterprise.v3.services.impl.SnifferAdapter.service(SnifferAdapter.java:178)
- locked <0xdbdce770> (a org.glassfish.internal.data.ContainerRegistry)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)


Under a normal operation,  it is the drainer thread (GFFileHandler) that process the logging. When the server.log is removed, it is seen that this thread also disappears. The drainer thread typically looks like this :

"Thread-2" prio=3 tid=0x09137400 nid=0x10 waiting on condition [0xce76c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdba063f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:514)
at com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:166)

 

Note: Stopping the GlassFish process using "asadmin stop-domain" will stall and time out. Hence to recover or to stop the process, the process needs to be killed by a process kill.

 

Changes

This symptom can happen only if the GlassFish server.log file is removed while GlassFish Server 3.1 is in started up and there is a thread writing extensively to the server.log file.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.