My Oracle Support Banner

EM13c: System Infrastructure Remote Host is in Status Pending or Showing Down in the EM Console. (Doc ID 2506986.1)

Last updated on JANUARY 30, 2020

Applies to:

Enterprise Manager for Exadata - Version 13.2.1.0.0 to 13.3.1.0.0 [Release 13c]
Information in this document applies to any platform.

Symptoms

After patching  Systems Infrastructure Remote Host targets are in Status Pending . Some times it shows DOWN in the console as well or Metric Collection error.

gcagent_sdk.trc:

2019-02-11 12:00:04,033 [271:BB3BB9AB] DEBUG - Session impl is oracle.sysman.si.agent.utils.pojo.sessionmgmt.impl.CiscoIOSSession
2019-02-11 12:00:04,033 [271:BB3BB9AB] DEBUG - Opening session to SessionIdentifier{uri=ios-ssh://mytest.server.com}
2019-02-11 12:00:04,131 [271:BB3BB9AB] DEBUG - JSCH LEVEL 2 : Permanently added 'mytest.server.com' (RSA) to the list of known hosts.
2019-02-11 12:00:04,135 [271:BB3BB9AB] DEBUG - promptKeyboardInteractive inputs : admin@mytest.server.com [Password: ] [false]
2019-02-11 12:00:04,907 [337:FCC194E6] DEBUG - ssh OpenConnection open throwing exception SSH_MSG_DISCONNECT: 2 Too many authentication failures for root
2019-02-11 12:00:04,907 [337:FCC194E6] DEBUG - jsch - throwing authentication exception for
com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 2 Too many authentication failures for root
at com.jcraft.jsch.Session.read(Session.java:1096)
at com.jcraft.jsch.UserAuthKeyboardInteractive.start(UserAuthKeyboardInteractive.java:77)
at com.jcraft.jsch.Session.connect(Session.java:478)
at oracle.sysman.si.agent.utils.pojo.sessionmgmt.impl.AbstractRemoteCommandSession.openSshSession(AbstractRemoteCommandSession.java:259)
at oracle.sysman.si.agent.utils.pojo.sessionmgmt.impl.AbstractRemoteCommandSession.openConnection(AbstractRemoteCommandSession.java:344)
at oracle.sysman.si.core.utils.agent.pojo.sessionmgmt.pojo.SessionManager.getSession(SessionManager.java:575)
at oracle.sysman.si.core.utils.agent.pojo.sessionmgmt.pojo.SessionManager.execCommandResult(SessionManager.java:284)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.execCommandResult(Exec.java:919)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.execCommandResult(Exec.java:614)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.execCommandResult(Exec.java:562)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:427)
at oracle.sysman.si.netswitch.ib.OracleInfiniBandSwitchSshHelper.platformCliExec(OracleInfiniBandSwitchSshHelper.java:626)
at oracle.sysman.si.netswitch.ib.OracleInfiniBandSwitchSshHelper.platformCliExec(OracleInfiniBandSwitchSshHelper.java:540)
at oracle.sysman.si.netswitch.ib.OracleInfiniBandSwitchSshHelper.platformCliExec(OracleInfiniBandSwitchSshHelper.java:497)
at oracle.sysman.si.netswitch.ib.OracleInfiniBandSwitchImpl.getResponse(OracleInfiniBandSwitchImpl.java:628)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.sysman.gcagent.addon.fetchlet.bean.BeanFetchlet.getMetric(BeanFetchlet.java:292)
at oracle.sysman.gcagent.target.interaction.execution.FetchletFactory.getMetric(FetchletFactory.java:426)
at oracle.sysman.gcagent.target.interaction.execution.ExecuteTask.executeQueryDescriptor(ExecuteTask.java:1213)

Other possible errors:

2019-01-30 17:34:49,048 [450:GC.DPExecutor2.7 (CDProps:oracle_si_server_os.vmy-test.oracle.com.com/ssh/mytest.oracle.com:3872/server_os:os.release)] ERROR - Failed to get OS name
oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Error: Authentication failure occurred executing command on target. Target: "ssh://mytest.server.com:22" Metric Collection: "Target Properties" Command: [/bin/uname -s] Timeout: 120 seconds.
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.execCommandResult(Exec.java:970)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:506)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:466)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:316)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:224)
at oracle.sysman.si.core.utils.agent.pojo.executor.Exec.exec(Exec.java:129)
at oracle.sysman.si.core.utils.agent.pojo.osutils.UnixOperatingSystemUtils.getOsName(UnixOperatingSystemUtils.java:162)
at oracle.sysman.si.core.os.impl.OsDynamicProperties.getOsName(OsDynamicProperties.java:228)
at oracle.sysman.si.core.os.impl.OsDynamicProperties.getOsRelease(OsDynamicProperties.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.sysman.gcagent.addon.fetchlet.bean.BeanFetchlet.populateRow(BeanFetchlet.java:487)
at oracle.sysman.gcagent.addon.fetchlet.bean.BeanFetchlet.getMetric(BeanFetchlet.java:331)
at oracle.sysman.gcagent.target.interaction.execution.FetchletFactory.getMetric(FetchletFactory.java:426)
at oracle.sysman.gcagent.target.interaction.execution.ExecuteTask.executeQueryDescriptor(ExecuteTask.java:1213)
at oracle.sysman.gcagent.target.interaction.execution.ExecuteTask.executeDPQueryDescriptor(ExecuteTask.java:997)
at oracle.sysman.gcagent.target.interaction.execution.ExecutePropertyComputation.call(ExecutePropertyComputation.java:941)
at oracle.sysman.gcagent.target.interaction.execution.ExecutePropertyComputation.call(ExecutePropertyComputation.java:67)
at oracle.sysman.gcagent.metadata.impl.DynamicProperties.computeProperties(DynamicProperties.java:213)
at oracle.sysman.gcagent.metadata.impl.TargetInstanceImpl$CalcDynamicPropsAction.call(TargetInstanceImpl.java:3917)

 

2018-09-18 13:09:12,234 INFO - Unschedule collections for target (exemptResponse=false, numOfCancelledCollections=0)
2018-09-18 13:09:12,243 INFO - Initialize collections for target
2018-09-18 13:09:12,244 INFO - Scheduling collections for target: 3
2018-09-18 13:09:12,319 INFO - Acknowledge upload of target dynamic properties: {os.release=UNSUPPORTED, os.name=UNSUPPORTED, os.server.uuid=<unknown>, os.virtual.platform.ldom=false, os.version=UNSUPPORTED, os.rad.modules.available=none, os.hostname=<unknown>, os.virtual.server=NO, os.virtual.platform.zone=false, os.virtual.server.name=<unknown>, os.xen.dom0=UNSUPPORTED, telemetry.dest.ip=0.0.0.0, __T_BROKENNESS_=0}

In the EM Console if we do see the following metric collection error:

 

From the Targets log we see the following errors in the logs:

2019-01-08 15:55:41,668 ERROR - Dynamic property execution error for [os.virtual.server], error: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}: java.lang.IllegalArgumentException: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}
2019-01-08 15:55:41,669 INFO - The target is (not fully) broken (code=0x4) for the following reason: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}: java.lang.IllegalArgumentException: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}
2019-01-08 15:55:41,674 INFO - Dynamic property execution error for [telemetry.dest.ip], error: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}: java.lang.IllegalArgumentException: Cannot create session SessionIdentifier{uri=ssh://mytest.server.com:22}
2019-01-08 15:55:41,674 INFO - Finished dynamic properties computation (total time 11 ms). Number of DP computed: 7. Number of DP with error: 2. Number of dynamic properties that were added: 5. Dynamic properties with changed value: [os.virtual.server.name, os.xen.dom0, os.hostname, os.server.uuid, os.rad.modules.available]. Dynamic properties that were loaded from the cache: []. Timeout=false
2019-01-08 15:55:41,674 ERROR - The target is fully broken (code=0x2004) for the following reason: Retries are completed.

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.