Unable To Start OBIEE 11g System Components Using Either Enterprise Manager Or OPMNCTL Commands (Doc ID 1300949.1)

Symptoms

When restarting the BI Components (OPMN), in a 2 server clustered environment, all services start on one server, but not on the other.

If the attempt to start the services, is executed at command line:

./ opmnctl startall

the following error is returned:

opmnctl startall: starting opmn and all managed processess...
Unacceptable request: OMPN is terminating
HTTP/1.1 406 Unacceptable request
Content-Length:  0
Content-Type:  text/html
Response: OPMN is terminating
/appl/fusion/mw_home/Oracle_BI1/opmn/bin/opmn:  unexpected exit: code 512
opmnctl startall: opmn failed to start.

Errors in the log files include:

1. AdminServer-diagnostic.txt:
   
   
   <Timestamp> [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: DmsThread-1] [userId: weblogic] [ecid: 0000Itn6tAC3V8q6wNJb6G1DRLJH00000P,1:17434] [APP: DMS Application#11.1.1.1.0] [dcid: 1a5f27d1e1e2bb48:2b0d4817:12e7313d303:-8000-0000000000000028] OPMN.queryProcessStatus failed with the following exception[[oracle.as.management.opmn.optic.OpticBadResponseException: Error while parsing OPMN dump in XML format: XML Parse Exception: tag=[null] Ran out of data, content []at oracle.as.management.opmn.optic.OpmnQuery.getDom(OpmnQuery.java:698)
   
   
     <Timestamp> [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: Thread-50] [userId: weblogic] [ecid: 0000Itn73mh3V8q6wNJb6G1DRLJH00000T,1:17442] [APP: NonJ2EEManagement#11.1.1] [dcid: 1a5f27d1e1e2bb48:2b0d4817:12e7313d303:-8000-000000000000002c] OPMN.queryProcessStatus failed with the following exception[[oracle.as.management.opmn.optic.OpticBadConnectException: Could not get secure socket for host <Hostname> and port 9501at oracle.as.management.opmn.optic.OpmnSSL.getSecureSocket(OpmnSSL.java:52)
2. OPMN.Log file:
   
   <Timestamp> [opmn][ERROR:1][17][OPMN][code:ons-listener] <IP Address>,9500: BIND (Address already in use)
   <Timestamp> [opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated|
   <Timestamp> [opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /appl/fusion_apps/instances/bipm_instance1/config/OPMN/opmn/states
   <Timestamp> [opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready. Request handling enabled.
   <Timestamp> [opmn][ERROR:1][17][OPMN][code:ons-listener]127.0.0.1,9500: BIND (Address already in use)

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.