CRS-2675: Stop of 'ora.oc4j' on 'racnode1' failed

(Doc ID 1586831.1)

Last updated on JULY 07, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

Stopping Grid Infrastructure with "crsctl stop crs", ora.oc4j fails to stop: 

CRS-2675: Stop of 'ora.oc4j' on 'racnode1' failed

 

<GI_HOME>/log/<node>/agent/crsd/scriptagent_oracle/scriptagent_oracle.log

2013-10-24 16:31:29.852: [ AGFW][10] {1:55506:30150} Agent received the message: RESOURCE_STOP[ora.oc4j 1 1] ID 4099:845527
2013-10-24 16:31:29.853: [ AGFW][10] {1:55506:30150} Preparing STOP command for: ora.oc4j 1 1
2013-10-24 16:31:29.853: [ AGFW][10] {1:55506:30150} ora.oc4j 1 1 state changed from: ONLINE to: STOPPING
2013-10-24 16:31:29.861: [ora.oc4j][11] {1:55506:30150} [stop] Executing action script: /opt/oracrs/gridsw/11203/bin/oc4jctl[stop]
2013-10-24 16:31:30.231: [ora.oc4j][11] {1:55506:30150} [stop] Stop OC4J
2013-10-24 16:31:46.907: [ora.oc4j][11] {1:55506:30150} [stop] Usage:
2013-10-24 16:31:46.907: [ora.oc4j][11] {1:55506:30150} [stop] ShutdownOc4j <url>
2013-10-24 16:31:46.907: [ora.oc4j][11] {1:55506:30150} [stop] Parameters:
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop] <url> -- The specific URL of the OC4J container.
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop] ormi[s]://<host>:<port>
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop] Other options:
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop] -h, -help -- Prints the ShutdownOc4j usage information
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop]
2013-10-24 16:31:46.908: [ora.oc4j][11] {1:55506:30150} [stop] Exception in thread "main" com.evermind.client.orion.AdminCommandException: Unexpected error during lookup : Lookup error: javax.naming.AuthenticationException: Not authorized; nested exception is:
2013-10-24 16:31:46.909: [ora.oc4j][11] {1:55506:30150} [stop] javax.naming.AuthenticationException: Not authorized
2013-10-24 16:31:46.909: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.client.orion.Oc4jAdminConsole.executeCommand(Oc4jAdminConsole.java:155)
2013-10-24 16:31:46.909: [ora.oc4j][11] {1:55506:30150} [stop] at oracle.cluster.wlm.dbwlm.consoleapps.ShutdownOc4j.shutdownOC4J(ShutdownOc4j.java:111)
2013-10-24 16:31:46.909: [ora.oc4j][11] {1:55506:30150} [stop] at oracle.cluster.wlm.dbwlm.consoleapps.ShutdownOc4j.main(ShutdownOc4j.java:67)
2013-10-24 16:31:46.910: [ora.oc4j][11] {1:55506:30150} [stop] Caused by: javax.naming.AuthenticationException: Not authorized
2013-10-24 16:31:46.910: [ora.oc4j][11] {1:55506:30150} [stop] at oracle.oc4j.rmi.ClientRmiTransport.connectToServer(ClientRmiTransport.java:100)
2013-10-24 16:31:46.910: [ora.oc4j][11] {1:55506:30150} [stop] at oracle.oc4j.rmi.ClientSocketRmiTransport.connectToServer(ClientSocketRmiTransport.java:69)
2013-10-24 16:31:46.910: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.server.rmi.RMIClientConnection.connect(RMIClientConnection.java:700)
2013-10-24 16:31:46.910: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.server.rmi.RMIClientConnection.sendLookupRequest(RMIClientConnection.java:244)
2013-10-24 16:31:46.911: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.server.rmi.RMIClientConnection.lookup(RMIClientConnection.java:228)
2013-10-24 16:31:46.911: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.server.rmi.RMIClient.lookup(RMIClient.java:302)
2013-10-24 16:31:46.911: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.server.rmi.RMIClientContext.lookup(RMIClientContext.java:59)
2013-10-24 16:31:46.911: [ora.oc4j][11] {1:55506:30150} [stop] at com.evermind.client.orion.Oc4jAdminConsole.executeCommand(Oc4jAdminConsole.java:138)
2013-10-24 16:31:46.911: [ora.oc4j][11] {1:55506:30150} [stop] ... 2 more
2013-10-24 16:32:04.205: [ AGFW][10] {1:55506:2} Agent received the message: AGENT_HB[Engine] ID 12293:845594
2013-10-24 16:33:04.207: [ AGFW][10] {1:55506:2} Agent received the message: AGENT_HB[Engine] ID 12293:845605
2013-10-24 16:33:16.219: [ora.oc4j][11] {1:55506:30150} [stop] Stop OC4J Failed
2013-10-24 16:33:16.220: [ AGFW][11] {1:55506:30150} Command: stop for resource: ora.oc4j 1 1 completed with status: FAIL
2013-10-24 16:33:16.221: [CLSFRAME][1] TM [MultiThread] is changing desired thread # to 4. Current # is 2
2013-10-24 16:33:16.223: [ora.oc4j][11] {1:55506:30150} [check] Executing action script: /opt/oracrs/gridsw/11203/bin/oc4jctl[check]
2013-10-24 16:33:16.223: [ AGFW][10] {1:55506:30150} Agent sending reply for: RESOURCE_STOP[ora.oc4j 1 1] ID 4099:845527
2013-10-24 16:33:16.366: [ora.oc4j][11] {1:55506:30150} [check] Check OC4J
2013-10-24 16:33:17.719: [ora.oc4j][11] {1:55506:30150} [check] http://localhost:8888/: HTTP::Response=HASH(0x1003f63b8)
2013-10-24 16:33:17.719: [ora.oc4j][11] {1:55506:30150} [check] Retcode (Container): 200
2013-10-24 16:33:17.719: [ora.oc4j][11] {1:55506:30150} [check] http://localhost:8888/dbwlm-root/dbwlm: HTTP::Response=HASH(0x100cc18b8)
2013-10-24 16:33:17.719: [ora.oc4j][11] {1:55506:30150} [check] Retcode (DBWLM): 200
2013-10-24 16:33:17.771: [ AGFW][10] {1:55506:30150} ora.oc4j 1 1 state changed from: STOPPING to: ONLINE
2013-10-24 16:33:17.771: [ AGFW][10] {1:55506:30150} Agent sending last reply for: RESOURCE_STOP[ora.oc4j 1 1] ID 4099:845527
2013-10-24 16:33:17.805: [ AGFW][10] {1:55506:30150} Agent received the message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:845633
2013-10-24 16:33:17.805: [ AGFW][10] {1:55506:30150} Preparing CLEAN command for: ora.oc4j 1 1
2013-10-24 16:33:17.805: [ AGFW][10] {1:55506:30150} ora.oc4j 1 1 state changed from: ONLINE to: CLEANING
2013-10-24 16:33:17.813: [ora.oc4j][15] {1:55506:30150} [clean] Executing action script: /opt/oracrs/gridsw/11203/bin/oc4jctl[clean]
2013-10-24 16:33:17.900: [ora.oc4j][15] {1:55506:30150} [clean] Clean OC4J
2013-10-24 16:33:18.083: [ AGFW][15] {1:55506:30150} Command: clean for resource: ora.oc4j 1 1 completed with status: SUCCESS
2013-10-24 16:33:18.117: [ AGFW][10] {1:55506:30150} Agent sending reply for: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:845633
2013-10-24 16:33:18.117: [ora.oc4j][14] {1:55506:30150} [check] Executing action script: /opt/oracrs/gridsw/11203/bin/oc4jctl[check]
2013-10-24 16:33:18.192: [ora.oc4j][14] {1:55506:30150} [check] Check OC4J
2013-10-24 16:33:18.243: [ AGFW][10] {1:55506:30150} ora.oc4j 1 1 state changed from: CLEANING to: OFFLINE
2013-10-24 16:33:18.244: [ AGFW][10] {1:55506:30150} Agent sending last reply for: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:845633

 

The oc4j resource fails to stop while running 'crsctl stop crs' causing stack stop failure on first node
crsctl stop crs -f works

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms