root.sh failed with "CLSRSC-1003: Failed to start resource OC4J" during 12c grid install. (Doc ID 1996831.1)

Last updated on JULY 02, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

1) Root.sh failed with:

CRS-2672: Attempting to start 'ora.asm' on 'hqxpl-orcl-d01'
CRS-2676: Start of 'ora.asm' on 'hqxpl-orcl-d01' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'hqxpl-orcl-d01'
CRS-2676: Start of 'ora.OCR.dg' on 'hqxpl-orcl-d01' succeeded
2015/02/11 14:15:38 CLSRSC-1003: Failed to start resource OC4J

 

2) In oc4j error log ($GRID_HOME/oc4j/j2ee/home/log), there are 2 main errors :

error1: [PersistentMap,$GRID_HOME/oc4j/j2ee/home/persistence/jms.state]: persistence file could not be locked

error2: port 23,792: Address already in use 

Mar 31, 2015 2:01:45 PM com.evermind.server.XMLApplicationServerConfig randomizeJtaAdminPassword
INFO: Updating JtaAdmin account
2015-03-31 14:01:45.783 ERROR ServerFile[PersistentMap,/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state]: persistence file could not be locked.  Resolve the sharing violation by ending the process currently holding the file lock or by using a different persistence directory for one or both processes.
15/03/31 14:01:45 *** (SEVERE) ServerFile[PersistentMap,/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state]: persistence file could not be locked.  Resolve the sharing violation by ending the process currently holding the file lock or by using a different persistence directory for one or both processes.
2015-03-31 14:01:45.786 ERROR Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
15/03/31 14:01:45 *** (SEVERE) Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
2015-03-31 14:01:45.788 ERROR J2EE OJR-00011 Exception starting JMS server: Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
2015-03-31 14:01:45.814 ERROR J2EE RMI-00002 Error starting ORMI server HQxPL-ORCL-D01 port 23,792: Address already in use
2015-03-31 14:01:45.815 ERROR java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.socketBind(PlainSocketImpl.java:521)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:414)
at java.net.ServerSocket.bind(ServerSocket.java:326)
at java.net.ServerSocket.<init>(ServerSocket.java:192)
at com.evermind.server.rmi.RMIServer.createServerSocket(RMIServer.java:459)
at com.evermind.server.rmi.RMIServer.createNonSecureSocket(RMIServer.java:373)
at com.evermind.server.rmi.RMIServer.setConfig(RMIServer.java:236)
at com.evermind.server.ApplicationServer.initializeRMI(ApplicationServer.java:2464)
at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1020)
at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:131)
at java.lang.Thread.run(Thread.java:682)

15/03/31 14:01:45 Error starting ORMI server HQxPL-ORCL-D01 port 23,792: Address already in use  

 

3) The file is actually locked the oc4j itself and the port 23792 also occupied by oc4j itself.

 

HQxPL-ORCL-D01:~ # netstat -anp|grep 23792
tcp        0      0 :::23792                :::*                    LISTEN      17510/java        

HQxPL-ORCL-D01:~ # ps -ef|grep 17510
grid     17510     1  0 Apr01 ?        00:02:54 /opt/oracle/app/12.1.0.2/grid/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -jar /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/oc4j.jar -config /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -properties -ports default-web-site:http:8888,rmi:23792 -out /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/log/oc4j.out -err /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/log/oc4j.err
root     34828 33106  0 17:27 pts/1    00:00:00 grep 17510

HQxPL-ORCL-D01:~ # lsof /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
java    17510 grid  165uw  REG  253,0     2048 638980 /opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state   ==>Occupied by process 17510.

Changes

11g RAC has been ever installed on those 2 nodes, but now deleted.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms