EM 12c: Enterprise Manager 12c Cloud Control OMS Recovery on New Server Fails with Error: Java.lang.Exception: REGISTER_WEBTIER_INST_FAILED (omsca.log) (Doc ID 2064993.1)

Last updated on APRIL 22, 2017

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Migrating an Enterprise Manager (EM) 12.1.0p.1 Cloud Control OMS to a new server using the 'omsca recover' command, fails with exception: java.lang.Exception: REGISTER_WEBTIER_INST_FAILED

Steps followed

1. Copy emkey to repository and Take backup of Primary OMS configuration

$OMS_HOME/bin/emctl config emkey -copy_to_repos

Example:

$OMS_HOME/bin/emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :XXXXXXX
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

$OMS_HOME/bin/emctl exportconfig oms -dir /backup/export

2. Shutdown primary OMS 

$OMS_HOME/bin/emctl stop oms -all

3. Copy backup (.bka) file to new server where recovery is to be performed

4. Recover OMS with -as option

$OMS_HOME/bin/omsca recover -as -ms -nostart -backup_file /backup/export/opf_ADMIN_20150815_024251.bka

Failing with following exception
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.4.0
Copyright (c) 1996, 2014, Oracle. All rights reserved.
    OS check passed.
    OMS version check passed.
    Performing Admin Server Recovery...
    Retrieved Admin Server template.
    Source Instance Host name where configuration is exported :uscdcodb02.oracle.com
    instHost name retrieved from local host name : uscdcodb06.oracle.com
    Populated install params from backup...
    Enter Administration Server user password:
    Confirm Password:
    Enter Node Manager Password:
    Confirm Password:
    Enter Repository database user password:
    Enter Agent Registration password:
    Confirm Password:
    Doing pre requisite checks ......
    Pre requisite checks completed successfully

    Checking Plugin software bits
    Proceed to recovery
    Setting up domain from template...
    Setup EM infrastructure succeeded!
    Admin Server recovered from backup.
    Now performing cleanup of OMS EMGC_OMS1...
    Now launching DeleteOMS...
    OMS Deleted successfully
    
    Delete finished successfully
    Now launching AddOMS...
    Infrastructure setup of EM completed successfully.

    Doing pre deployment operations ......
    Pre deployment of EM completed successfully.

    Deploying EM ......
    Deployment of EM completed successfully.

    Configuring webtier ......
    java.lang.Exception: REGISTER_WEBTIER_INST_FAILED
     at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.adapterConfigWebTier(OMSWLSAdapter.java:2313)
     at oracle.sysman.omsca.framework.OMSGenericAdapter.configWebTier(OMSGenericAdapter.java:312)
     at oracle.sysman.omsca.framework.OMSCARecoveryConfig.execAddOMS(OMSCARecoveryConfig.java:997)
     at oracle.sysman.omsca.framework.OMSCARecoveryConfig.execute(OMSCARecoveryConfig.java:778)
     at oracle.sysman.omsca.framework.OMSConfigAssistantDriver.main(OMSConfigAssistantDriver.java:240)
    OMSCA-ERR:Configuring WebTier failed
    OMSCA-ERR:Configuring WebTier failed.
    Check the trace file:/app/oracle/product/12.1.0.4/middleware/oms/cfgtoollogs/omsca/omsca_20150815073225.log
    OMSCA-ERR:Configuring WebTier failed. Check the trace  file:/app/oracle/product/12.1.0.4/middleware/oms/cfgtoollogs/omsca/omsca_20150815073225.log
    Add OMS failed
    OMS CA recover failed   

/OMS_HOME/cfgtoollogs/omsca_20150815032653.log reports following exception

Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.adapter.wls.OMSWLSAdapter adapterCleanup
FINE: The value of infra setup completion is :true
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.adapter.wls.OMSWLSAdapter adapterConfigWebTier
SEVERE: Exception occurred while configuring WebTier:
java.lang.Exception: REGISTER_WEBTIER_INST_FAILED
  at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.adapterConfigWebTier(OMSWLSAdapter.java:2313)
  at oracle.sysman.omsca.framework.OMSGenericAdapter.configWebTier(OMSGenericAdapter.java:312)
  at oracle.sysman.omsca.framework.OMSCARecoveryConfig.execAddOMS(OMSCARecoveryConfig.java:997)
  at oracle.sysman.omsca.framework.OMSCARecoveryConfig.execute(OMSCARecoveryConfig.java:778)
  at oracle.sysman.omsca.framework.OMSConfigAssistantDriver.main(OMSConfigAssistantDriver.java:240)
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.framework.OMSGenericAdapter configWebTier
SEVERE: OMSCA-ERR:Configuring WebTier failed. Check the trace file:/app/oracle/product/12.1.0.4/middleware/oms/cfgtoollogs/omsca/omsca_20150815032653.log
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.framework.OMSCARecoveryConfig execAddOMS
SEVERE: Configuring of WebTier failed.
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.framework.OMSCARecoveryConfig execute
SEVERE: Add OMS failed
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.framework.OMSCARecoveryConfig execute
FINE: Performing cleanup...
Aug 15, 2015 3:39:40 AM oracle.sysman.omsca.framework.OMSConfigAssistantDriver main
FINE: Got resultfalse
"omsca_20150815032653.log" 2617L, 202135C

/OMS_HOME/cfgtoollogs/omsca/logs20150815073225/webtier/diagnostics/logs/OPMN/opmn/provision.log reports following error

Aug 15, 2015 7:36:08 AM oracle.as.config.provisioner.InstallerCmdLine logInitialCommand
INFO: Start of command: [unregisterinstance, -oracleInstance, /app/oracle/product/12.1.0.4/gc_inst/WebTierIH1, -adminHost, uscdcodb06.us.cbre.net, -adminPort, 7101, -adminProtocol, t3s, -adminUsername, weblogic, -adminPasswordFile, /app/oracle/product/12.1.0.4/gc_inst/WebTierIH1/adminPwd.txt.delete_this, -logLevel, FINEST, -instanceName, instance1]
...............
Aug 15, 2015 7:40:10 AM oracle.as.config.provisioner.InstallerCmdLine logInitialCommand
INFO: Start of command: [registerinstance, -oracleInstance, /app/oracle/product/12.1.0.4/gc_inst/WebTierIH1, -instanceName, instance1, -adminHost, uscdcodb06.us.cbre.net, -adminPort, 7101, -adminProtocol, t3s, -adminUsername, weblogic, -adminPasswordFile, /app/oracle/product/12.1.0.4/gc_inst/WebTierIH1/adminPwd.txt.delete_this, -logLevel, FINEST]
......................
Aug 15, 2015 7:40:10 AM oracle.as.config.impl.RuntimeServiceConnectionWLS connect
FINER: Creating connection to hello there weblogic.management.mbeanservers.runtime
Aug 15, 2015 7:40:10 AM oracle.as.config.impl.RuntimeServiceConnectionWLS connect
FINER: Setting (temporary) security properties
Aug 15, 2015 7:45:39 AM oracle.as.config.provisioner.InstallerCmdLine handleRunFailure
SEVERE: Command failed:
oracle.as.config.ProvisionException: Unable to connect to the admin server.  Verify it has been started and is accessible.
at oracle.as.config.impl.RuntimeServiceConnectionWLS.connect(RuntimeServiceConnectionWLS.java:138)
at oracle.as.config.impl.OracleASInstanceImpl.register(OracleASInstanceImpl.java:282)
at oracle.as.config.provisioner.commands.RegisterInstanceCommand.execute(RegisterInstanceCommand.java:11)
at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: java.io.IOException: Unable to connect to the server.  Verify it has been started and is accessible.
at oracle.as.management.mbeans.util.MBeanClient.connect(MBeanClient.java:122)
at oracle.as.config.impl.RuntimeServiceConnectionWLS.connect(RuntimeServiceConnectionWLS.java:121)
... 4 more
Caused by: java.io.IOException
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:196)
at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:84)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:338)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:247)
at oracle.as.management.mbeans.util.MBeanClient.connect(MBeanClient.java:118)
... 5 more
Caused by: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://uscdcodb06.oracle.com:7101: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366)
at weblogic.jndi.Environment.getContext(Environment.java:315)
at weblogic.jndi.Environment.getContext(Environment.java:285)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
at javax.naming.InitialContext.init(InitialContext.java:223)
at javax.naming.InitialContext.<init>(InitialContext.java:197)
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:178)
... 9 more
Caused by: java.net.ConnectException: t3s://uscdcodb06.uscdcodb06.oracle.com:7101: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340)
... 17 more
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms