In 11.2 and 12c, stopping ons fails. The rootupgrade.sh, "crsctl stop crs -f", and "srvctl stop nodeapps" fail because ons fails to stop.

(Doc ID 2038830.1)

Last updated on AUGUST 21, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 to 12.1.0.2 [Release 11.2 to 12.1]
Information in this document applies to any platform.

Symptoms

1) rootupgrade.sh fails on the first node because it could not stop CRS.

The output shows that it failed to stop ons

PRCR-1014 : Failed to stop resource ora.ons
PRCR-1065 : Failed to stop resource ora.ons
CRS-5014: Agent "/opt/grid/11.2.0.4/bin/oraagent.bin" timed out starting process "/opt/grid/11.2.0.4/opmn/bin/onsctli" for action "clean": details at "(:CLSN00009:)" in "/opt/grid/11.2.0.4/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-5017: The resource action "ora.ons clean" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00106:)" in "/opt/grid/11.2.0.4/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log".

CRS-5014: Agent "/opt/grid/11.2.0.4/bin/oraagent.bin" timed out starting process "/opt/grid/11.2.0.4/opmn/bin/onsctli" for action "check": details at "(:CLSN00009:)" in "/opt/grid/11.2.0.4/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-5017: The resource action "ora.ons check" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00109:)" in "/opt/grid/11.2.0.4/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log".

CRS-2680: Clean of 'ora.ons' on 'racnode1' failed
PRKO-2425 : VIP is already stopped on node(s): racnode1
PRKO-2440 : Network resource is already stopped.

 

2) "srvctl stop nodeapps" fails because ons can not be stopped

CRS-2680: Clean of 'ora.ons' on 'racnode1' failed
PRKO-2425 : VIP is already stopped on node(s): racnode1
PRKO-2440 : Network resource is already stopped

3) "crsctl stop crs" and "crsctl stop crs -f" fail because ons can not be stopped

4) "GI_HOME/opmn/bin/onsctli ping" and "GI_HOME/opmn/bin/onsctli shutdown" fails with the following error.

Communication error with the OPMN server local port.
Check the OPMN log files
......
Communication error with the OPMN server local port.
Check the OPMN log files

onsctl shutdown: ons is not running.

5) "ps -ef | grep ons" show two ons processes (expected)

oracle    2165         1  0 Jul22 ?        00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
oracle    2166  2165  0 Jul22 ?        00:00:34 /u01/app/11.2.0/grid/opmn/bin/ons -d

However, "GI_HOME/opmn/bin/onsctli ping" fails with the following error.

Communication error with the OPMN server local port.
Check the OPMN log files
......
Communication error with the OPMN server local port.
Check the OPMN log files

onsctl shutdown: ons is not running.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms