JMQ Consumer Process Takes Too Long To Shutdown On High Load ISS Production Servers (Doc ID 2086375.1)

Last updated on DECEMBER 18, 2023

Applies to:

Symptoms

When executing svc_control to stop the ISS services on a high load production server, it takes an inordinate amount of time to for the process to shutdown.

Is there a timeout value that can be adjusted or a way to kill the jmq consumer? It's possible to get in a situation where someone is frustrated after a long wait and will do a kill -9 on all jiss processes, leaving us vulnerable to index corruption because the processes did not come down cleanly.

Example:

During peak hours, the following server took around 9 minutes for the everything to shutdown (jmqconsumer took about 6.5 mins - see below):

(23:18:31) root@iss-host [~] /san/jiss/bin/svc_control.sh stop
Stopping watcher
Stopping jmqconsumer
Stopping com.sun.comms.iss.jmqconsumer.JMQConsumer
PID: 8372
Checking pid: 8372
Pid 8372 is still alive.
Checking pid: 8372
Pid 8372 is still alive.
.....
Checking pid: 8372
Pid 8372 is still alive.
Checking pid: 8372
Pid 8372 is still alive.
Checking pid: 8372
Pid 8372 is still alive.
.....
Stopping searchSvc
Stopping indexSvc
Stopping utilSvc
(23:27:35) root@iss-host [~]

Load average wasn't significant:

(23:27:35) root@st11p01mm-iss010206 [~] uptime
11:28pm up 234 day(s), 6:23, 1 user, load average: 0.71, 2.12, 2.35

Several index processes were orphaned after the shutdown which seems typical for this server:

(23:28:45) root@iss-host [~] ps -fu jiss
UID PID PPID C STIME TTY TIME CMD
jiss 28297 28296 0 09:00:01 ? 0:26 /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexAdmin -
jiss 10705 10704 0 21:00:01 ? 0:07 /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexAdmin -
jiss 19151 19114 0 Nov 15 ? 0:00 sh -c /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexA
jiss 21427 21400 0 Nov 15 ? 0:00 sh -c /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexA
jiss 21428 21427 0 Nov 15 ? 0:51 /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexAdmin -
jiss 10704 10677 0 21:00:01 ? 0:00 sh -c /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexA
jiss 3802 3801 0 15:00:01 ? 0:17 /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexAdmin -
jiss 28296 28269 0 09:00:01 ? 0:00 sh -c /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexA
jiss 19152 19151 0 Nov 15 ? 1:04 /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexAdmin -
jiss 3801 3774 0 15:00:01 ? 0:00 sh -c /usr/jdk/latest/bin/amd64/java -Xmx512m com.sun.comms.iss.indexapi.IndexA

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.