EM 12c, EM 13c: Enterprise Manager Cloud Control Agent Status Command Fails with Message: Agent Unreachable (REASON = unable to connect to http server [peer not authenticated]) (Doc ID 1919204.1)

Last updated on APRIL 28, 2023

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Enterprise Manager (EM) 12c Cloud Control Agent status fails with "Agent unreachable" error:

Use Case 1 (Single OMS environment)

$AGENT_INST/bin/emctl status agent

fails with following message :

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Status agent Failure:unable to connect to http server at https://myagent.oracle.com:3872/emd/lifecycle/main/. [peer not authenticated]

A notification may also have triggered with 'Agent Unreachable (REASON = unable to connect to http server [peer not authenticated])'

Use Case 2 (Multi OMS environment)

Certificate serial# mismatch in multi OMS environment.

gc_inst/em/EMGC_OMS1/sysman/log/emoms.trc reports following message:

WARN emSDK.comm setInstanceReadTimeout.10588 - unable to set the instance read timeout: unable to connect to http server at https://myagent.oracle.com:3872/emd/main/. [peer not authenticated]

Verifying certificate details at each OMS shows Serial# mismatch.

1. Certificate dump from OMS1 server

$OMS_HOME/bin/emctl secdiag dumpcertsinfile -file gc_inst/em/EMGC_OMS1/sysman/config/b64LocalCertificate.txt > /tmp/output_OMS1.txt

2. Certificate dump from OMS2 server

$OMS_HOME/bin/emctl secdiag dumpcertsinfile -file gc_inst/em/EMGC_OMS2/sysman/config/b64LocalCertificate.txt > /tmp/output_OMS2.txt

3. Certificate dumps shows Serial# mismatch

$ cat /tmp/output_OMS1.txt

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

Log file: /tmp/DumpCertsInFile_2014_10_08_11_50_292629879523546025234.log

Cert#1:
Subject: CN=hostname.com, C=US, ST=CA, L=EnterpriseManager on hostname.com, OU=EnterpriseManager on hostname.com, O=EnterpriseManager on hostname.com
Issuer: CN=hostname.com, C=US, ST=CA, L=EnterpriseManager on hostname.com, OU=EnterpriseManager on hostname.com, O=EnterpriseManager on hostname.com
Valid from: Tue Jan 03 17:03:14 MST 2012
Valid till: Sat Jan 01 17:03:14 MST 2022
Serial#: 89447815409407087090
Public Key: Sun RSA public key, 512 bits
public exponent: 65537
Signature algorithm: MD5withRSA

Cert#2:
Subject: CN=hostname.com, OU=EnterpriseManager, O=EnterpriseManager, L=Safeway, ST=CA, C=US
Issuer: CN=hostname.com, OU=oracle.com, O=safeway.com, L=Phoenix, ST=Arizona, C=US
Valid from: Mon Nov 04 01:00:00 MST 2013
Valid till: Sun Nov 04 00:59:59 MDT 2018
Serial#: 990
Public Key: Sun RSA public key, 2048 bits

Cert#3:
Subject: CN=hostname.com, OU=oracle.com, O=hostname.com, L=Phoenix, ST=Arizona, C=US
Issuer: CN=hostname.com, OU=oracle.com, O=hostname.com, L=Phoenix, ST=Arizona, C=US
Valid from: Thu Dec 02 00:00:00 MST 2010
Valid till: Sun May 06 00:59:59 MDT 2035
Serial#: 0
Public Key: Sun RSA public key, 1024 bits public exponent: 65537
Signature algorithm: SHA1withRSA

** and **

$ cat /tmp/output_OMS2.txt

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

Log file: /tmp/DumpCertsInFile_2014_10_08_11_51_217555904462707659154.log

Cert#1:
Subject: CN=hostname.com, OU=EnterpriseManager, O=EnterpriseManager, L=Safeway, ST=CA, C=US
Issuer: CN=hostname.com, OU=oracle.com, O=oracle.com, L=Phoenix, ST=Arizona, C=US
Valid from: Mon Nov 04 01:00:00 MST 2013
Valid till: Sun Nov 04 00:59:59 MDT 2018
Serial#: 990
Public Key: Sun RSA public key, 2048 bits
public exponent: 65537
Signature algorithm: SHA1withRSA

Cert#2:
Subject: CN=hostname.com, OU=oracle.com, O=oracle.com, L=Phoenix, ST=Arizona, C=US
Issuer: CN=hostname.com, OU=oracle.com, O=oracle.com, L=Phoenix, ST=Arizona, C=US
Valid from: Thu Dec 02 00:00:00 MST 2010
Valid till: Sun May 06 00:59:59 MDT 2035
Serial#: 0
Public Key: Sun RSA public key, 1024 bits
public exponent: 65537
Signature algorithm: SHA1withRSA

From the above details, it can be seen that OMS1 has certificate with Serial#: 89447815409407087090 which is missing on the OMS2.

Use Case 3

A notification may also have triggered with 'Message=Agent is Unreachable (REASON = unable to connect to http server at https://<HOSTNAME>.<DOMAIN_NAME>:3872/emd/main/. [peer not authenticated]) but the host is reachable.

/AGENT_INST/sysman/log/emctl.log eports agent hung
28519 :: Wed Nov 5 12:50:41 2014::Blackout oper: start, Name: blackout_IDMPROOF : Released lock
28519 :: Wed Nov 5 12:50:41 2014::AgentStatus.pm:emdctl start blackout blackout_IDMPROOF IDMPROOF:oracle_database -d 240 returned 1
2176 :: Wed Nov 5 13:40:10 2014::AgentLifeCycle.pm: Processing stop agent

No entry betweem 12.50 to 13.40 on 5th Nov

/AGENT_INST/sysman/log/emdctlj.log reports connection exception

2014-11-05 13:40:14,881 [main] INFO - unable to connect to the agent at https://<HOSTNAME>.<DOMAIN_NAME>:3872/emd/main/ [Connection refused]
oracle.sysman.emSDK.agent.comm.exception.ConnectException: unable to connect to the agent at https://<HOSTNAME>.<DOMAIN_NAME>:3872/emd/main/ [Connection refused]
at oracle.sysman.gcagent.comm.oms.http.TMClientConnection.newConnectException(TMClientConnection.java:231)
at oracle.sysman.gcagent.comm.http.ClientConnection.makeConnection(ClientConnection.java:820)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)