Intermittent Hang Of Web Cache And Automatic Restart By OPMN

Last updated on APRIL 09, 2018

Applies to:

Symptoms

Web Cache restarts unexpectedly.

This is verified in the Web Cache event_log which shows (snippet) -

[2010-06-22T10:32:05+02:00] [webcache] [TRACE:32] [WXE-13605] [main] [ecid: ] Signal SIGTERM caught
[2010-06-22T10:32:05+02:00] [webcache] [NOTIFICATION:1] [WXE-09703] [main] [ecid: ] Stop Issued. The program will shut down after all accepted requests are served, or a timeout occurs.
.
.
[2010-06-22T10:32:06+02:00] [webcache] [NOTIFICATION:1] [WXE-08514] [logging] [ecid: ] Cache server process ID 32110 is shutting down.
[2010-06-22T10:32:43+02:00] [webcache] [NOTIFICATION:1] [WXE-08513] [logging] [ecid: ] Cache server process ID 19508 is starting up.
[2010-06-22T10:32:43+02:00] [webcache] [NOTIFICATION:1] [WXE-09612] [main] [ecid: ] Oracle Web Cache 11g (11.1.1), Build 11.1.1.1.0 090603.1900
[2010-06-22T10:32:43+02:00] [webcache] [NOTIFICATION:1] [WXE-09403] [main] [ecid: ] Maximum number of file/socket descriptors set to 5945.
[2010-06-22T10:32:43+02:00] [webcache] [NOTIFICATION:1] [WXE-13002] [config] [ecid: ] Maximum allowed incoming connections are 2000
.
.
[2010-06-22T11:23:55+02:00] [webcache] [TRACE:32] [WXE-13605] [main] [ecid: ] Signal SIGTERM caught
.
.
[2010-06-22T11:41:48+02:00] [webcache] [TRACE:32] [WXE-13605] [main] [ecid: ] Signal SIGTERM caught
.
.

At the same time, the console_WebCache_1.log shows the standard restart message:
10/06/22 10:32:05 Stop process
Web Cache auto-restart monitor signaled to stop.
Web Cache cache server signaled to stop.

The OPMN log files opmn.log and debug.log show a ping timout error:
eg
[2010-06-24T11:27:33][opmn][TRACE:1][][OPMN][code:libopmnwc]Process Ping Failed: webcache1~WebCache~WebCache~1 (757285066:8215) [The connection receive timed out]
[2010-06-24T11:28:23][opmn][TRACE:1][][OPMN][code:libopmnwc]Process Ping Failed: webcache1~WebCache~WebCache~1 (757285066:8215) [The connection receive timed out]
[2010-06-24T11:29:13][opmn][TRACE:1][][OPMN][code:libopmnwc]Process Ping Failed: webcache1~WebCache~WebCache~1 (757285066:8215) [The connection receive timed out]
[2010-06-24T11:29:13][opmn][TRACE:1][][OPMN][code:libopmnwc]Process Unreachable: webcache1~WebCache~WebCache~1 (757285066:8215)
[2010-06-24T11:29:13][opmn][TRACE:1][663][OPMN][code:pm-process]Stopping Process: webcache1~WebCache~WebCache~1 (757285066:8215)
[2010-06-24T11:30:13][opmn][TRACE:1][663][OPMN][code:pm-process]Stopping Process: webcache1~WebCache~WebCache~1 (757285066:8215)
[2010-06-24T11:30:13][opmn][TRACE:1][][OPMN][code:libopmnwc]Forcefully Terminating Process: webcache1~WebCache~WebCache~1 (757285066:8215)
[2010-06-24T11:30:15][opmn][TRACE:1][662][OPMN][code:pm-process]Starting Process: webcache1~WebCache~WebCache~1 (757285066:0)

If Web Cache is set to use the more detailed logging level of Trace, then messages showing  which look like the following may bee seen in the event_log (the detailed log level of Trace has the overhead of producing a very large log file):
[Thu Sep 10 07:29:14 2009] [event log] [Open] [HeaderBad]  [/u01/app/oracle/product/instances/WT11gR1/diagnostics/logs/WebCache/webcache_ucm/event_log]


Ensure there are no cron jobs which are configured to impose a restart, which would result in the same symptoms.

Cause