ONS STOP FAILURE REPORTED DURING GI STACK SHUTDOWN (Doc ID 2130351.1)

Last updated on JUNE 30, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

ONS stop failure reported during GI stack shutdown:


CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '<node1>'
CRS-2673: Attempting to stop 'ora.crsd' on '<node1>'
....
CRS-2673: Attempting to stop 'ora.ons' on '<node1>'
CRS-5014: Agent "ORAAGENT" timed out starting process "/home/app/12.2.0/grid/opmn/bin/onsctli" for action "stop": details at "(:CLSN00009:)" in "/home/app/oracle/diag/crs/<node1>/crs/trace/crsd_oraagent_crsusr.trc"
CRS-5017: The resource action "ora.ons stop" encountered the following error:
 
(:CLSN00009:)Command Aborted. For details refer to "(:CLSN00108:)" in "/home/app/oracle/diag/crs/<node1>/crs/trace/crsd_oraagent_crsusr.trc".
CRS-2675: Stop of 'ora.ons' on '<node1>' failed <== here
 
CRS-2679: Attempting to clean 'ora.ons' on '<node1>'
CRS-2681: Clean of 'ora.ons' on '<node1>' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on '<node1>'
CRS-2677: Stop of 'ora.net1.network' on '<node1>' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on '<node1>' has completed
CRS-2677: Stop of 'ora.crsd' on '<node1>' succeeded

From crsd oraagent log on node <node1>:

2014-11-25 22:43:34.054099 :    AGFW:4018124544: {10:15575:530} Preparing STOP command for: ora.ons <node1> 1
2014-11-25 22:43:34.054110 :    AGFW:4018124544: {10:15575:530} ora.ons <node1> 1 state changed from: ONLINE to: STOPPING
2014-11-25 22:43:34.054776 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] (:CLSN00108:) clsn_agent::stop {
2014-11-25 22:43:34.054824 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgent::stop {
2014-11-25 22:43:34.054867 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
2014-11-25 22:43:35.033774 : USRTHRD:4016023296:  ONS OnsEventForwarder::postOnsEvent posting event "VERSION=1.0 event_type=NODE host=hi07-1f incarn=0 status=nodedown reason=public_nw_down vip_ips=10.196.7.121 timestamp=2014-11-25 22:43:34 timezone=-08:00"
2014-11-25 22:43:35.054981 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
2014-11-25 22:43:35.548299 : USRTHRD:4016023296:  CLSN.ONSPROC.MASTER CssSemaphore::tryAcquire, acquired semaphore CLSN.ONSPROC.MASTER
2014-11-25 22:43:35.548327 : USRTHRD:4016023296:  ONSPROC CssSemMM::tryMaster I am the master
2014-11-25 22:43:35.559228 : USRTHRD:4016023296:  CssSemaphore::release, released semaphore CLSN.ONSNETPROC.MASTER
2014-11-25 22:43:35.559264 : USRTHRD:4016023296:  ONSNETPROC CssSemMM::relinquishMastership I am no longer the master
2014-11-25 22:43:35.559496 : USRTHRD:4016023296:  Processing the event CRS_RESOURCE_STATE_CHANGE
2014-11-25 22:43:35.560016 : USRTHRD:4016023296:  Processing the event NO-OP
2014-11-25 22:43:35.560565 : USRTHRD:4016023296:  Processing the event NO-OP
2014-11-25 22:43:36.055103 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
....
2014-11-25 22:44:31.061379 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
2014-11-25 22:44:32.061481 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
2014-11-25 22:44:33.061595 :CLSDYNAM:4002416384: [ ora.ons]{10:15575:530} [stop] OnsAgentEventModule::stop waiting for EventBridge to shut down
2014-11-25 22:44:34.056203 :   AGENT:3758094080: {10:15575:530} {10:15575:530} Created alert : (:CRSAGF00113:) :  Aborting the command: stop for resource: ora.ons <node1> 1
2014-11-25 22:44:34.056234 :CLSDYNAM:3758094080: [ ora.ons]{10:15575:530} [stop] (:CLSN00110:) clsn_agent::abort {
2014-11-25 22:44:34.056265 :CLSDYNAM:3758094080: [ ora.ons]{10:15575:530} [stop] abort {
2014-11-25 22:44:34.056281 :CLSDYNAM:3758094080: [ ora.ons]{10:15575:530} [stop] Default Agent Dump
2014-11-25 22:44:34.056289 :CLSDYNAM:3758094080: [ ora.ons]{10:15575:530} [stop] Agent::setResAttrib traces last call info:
2014-11-25 22:44:34.056303 :CLSDYNAM:3758094080: [ ora.ons]{10:15575:530}[stop]  Time:11/25/2014 19:56:22.641 Tint:{10:15575:2} action:101 resname:ora.ons lastCall:Agent::Agent refreshAttr

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms