My Oracle Support Banner

A node in a RAC cluster was unexpectedly rebooted, with captured ps output showing csssdagent and other processes in futex_ state immediately prior to reboot (Doc ID 2399800.1)

Last updated on AUGUST 04, 2018

Applies to:

Linux OS - Version Oracle Linux 5.11 with Unbreakable Enterprise Kernel [2.6.39] and later
Information in this document applies to any platform.

Symptoms

A Linux server which was part of a RAC cluster was unexpectedly rebooted.
An oswatcher archive was available covering the time period immediately before the reboot, and it shows multiple processes all stuck in a futex lock:

4 S oragrid 8790 1 0 75 0 - 114422 futex_ 2017 ? 04:31:22 /data/oragrid/product/11.2.0.4.1/bin/oraagent.bin
0 S oracle 13917 13755 0 78 0 - 299320 futex_ 2017 ? 01:27:02 /data/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/java -Xmx256M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=13755 -cp /data/oracle/product/agent12c/core/12.1.0.2.0/jdbc/lib/ojdbc5.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/ucp/lib/ucp.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.http_client_11.1.1.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/xmlparserv2.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/jsch.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/optic.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.dms_11.1.1/dms.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.odl_11.1.1/ojdl.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.odl_11.1.1/ojdl2.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/log4j-core.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/jlib/gcagent_core.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/emagentSDK-intg.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain
4 S root 17445 1 0 -40 - - 64464 futex_ 2017 ? 00:21:52 /data/oragrid/product/11.2.0.4.1/bin/cssdmonitor
4 S root 17456 1 0 -40 - - 64510 futex_ 2017 ? 00:14:39 /data/oragrid/product/11.2.0.4.1/bin/cssdagent
4 S oragrid 17468 1 1 -40 - - 93317 futex_ 2017 ? 10:48:01 /data/oragrid/product/11.2.0.4.1/bin/ocssd.bin
4 S root 18382 1 0 75 0 - 77566 futex_ 2017 ? 00:07:25 /data/oragrid/product/11.2.0.4.1/bin/octssd.bin
5 S root 32009 1 2 -40 - - 54343 futex_ 2017 ? 2-13:21:48 /sbin/multipathd
4 S root 1410 1 0 75 0 - 91030 futex_ 16:47 ? 00:00:01 /data/oragrid/product/11.2.0.4.1/bin/orarootagent.bin
4 S root 2836 2834 0 67 -8 - 37061 futex_ 2017 ? 00:01:58 /sbin/audispd
4 S oracle 4578 1 0 75 0 - 137246 futex_ 2017 ? 00:28:57 /data/oragrid/product/11.2.0.4.1/bin/scriptagent.bin
5 S root 4703 1 0 80 0 - 3978 futex_ 2017 ? 03:16:07 /opt/imperva/ragent/bin/ragentwd --daemon -r /opt/imperva/ragent -d /opt/imperva/ragent/etc -v /opt/imperva/ragent/var
4 S root 4716 4703 0 82 0 - 5663 futex_ 2017 ? 03:35:03 /opt/imperva/ragent/bin/ragentctrl -r /opt/imperva/ragent -d /opt/imperva/ragent/etc -v /opt/imperva/ragent/var -t ragent --port 50101
4 S root 4738 4703 0 -40 - - 24449 futex_ 2017 ? 08:23:34 /opt/imperva/ragent/bin/ragent -r /opt/imperva/ragent/ -d /opt/imperva/ragent/etc -v /opt/imperva/ragent/var --ncpu 1 --port 50102
4 S oragrid 8790 1 0 75 0 - 114422 futex_ 2017 ? 04:31:22 /data/oragrid/product/11.2.0.4.1/bin/oraagent.bin
0 S oracle 13917 13755 0 78 0 - 299320 futex_ 2017 ? 01:27:02 /data/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/java -Xmx256M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=13755 -cp /data/oracle/product/agent12c/core/12.1.0.2.0/jdbc/lib/ojdbc5.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/ucp/lib/ucp.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.http_client_11.1.1.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/xmlparserv2.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/jsch.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/lib/optic.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.dms_11.1.1/dms.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.odl_11.1.1/ojdl.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/modules/oracle.odl_11.1.1/ojdl2.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/log4j-core.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/jlib/gcagent_core.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/emagentSDK-intg.jar:/data/oracle/product/agent12c/core/12.1.0.2.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain
4 S root 17445 1 0 -40 - - 64464 futex_ 2017 ? 00:21:53 /data/oragrid/product/11.2.0.4.1/bin/cssdmonitor
4 S root 17456 1 0 -40 - - 64510 futex_ 2017 ? 00:14:39 /data/oragrid/product/11.2.0.4.1/bin/cssdagent
4 S oragrid 17468 1 1 -40 - - 93317 futex_ 2017 ? 10:48:01 /data/oragrid/product/11.2.0.4.1/bin/ocssd.bin
4 S root 18382 1 0 75 0 - 77566 futex_ 2017 ? 00:07:25 /data/oragrid/product/11.2.0.4.1/bin/octssd.bin

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.