11g Grid Control: Lot of Stuck WLS Threads Related to Log Rotation Affecting OMS Performance (Doc ID 1327131.1)

Last updated on OCTOBER 15, 2014

Applies to:

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Oracle Weblogic Server - Version 10.3.2 to 10.3.2
Information in this document applies to any platform.

Symptoms

Using 11g OMS Setup. A lot of stuck threads related with the process of rotating files were found in the EMGC_OMS.log file:

<May 24, 2011 1:04:42 AM PDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '129' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1c43e81[
POST /em/upload?type=uploadonfetch&EMD_URL=https%3a%2f%2fusto-tdbx-ora15%2eamgen%2ecom%3a1830%2femd%2fmain%2f&X-ORCL-EMOV=4%2e0%2e0&X-ORCL-EMCV=10%2e2%2e0%2e5%2e0&X-ORCL-EMSV=10%2e2%2e0%2e5%2e0 HTTP/1.1
X-ORCL-EMUR: https://agentmachine.domain:1830/emd/main/
X-ORCL-EMAK: 73624282325B91CC8D86A43F005E7E45
Content-Length: 7806
ECID-Context: 1.004d03aYjvQCwkupO^T4id0007f10000JF;kXjE1ZDLIPHGj5PRj5SSgVPO^G
Connection: Keep-Alive
Proxy-Client-IP: <IP>
X-Forwarded-For: <IP>
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -1644762696

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
com.bea.logging.RotatingFileOutputStream.write(RotatingFileOutputStream.java:228)
weblogic.servlet.logging.CLFLogger.log(CLFLogger.java:108)
weblogic.servlet.logging.LogManagerHttp.log(LogManagerHttp.java:121)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1510)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1441)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>


For the examined system, every 20 minutes a new access log file was created, so log directory had 72 additional files per day. If no maintenance is made to log directory or WLS logging setting, even log directory could hold thousand of files, generating a potential performance issue.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms