EM 12c: OHS Target Is Intermittently Shown As Down In Enterprise Manager 12c Cloud Control (Doc ID 1505506.1)

Last updated on DECEMBER 13, 2016

Applies to:

Enterprise Manager for Fusion Middleware - Version 12.1.0.1.0 to 12.1.0.3.0
Information in this document applies to any platform.

Symptoms

Using Enterprise Manager Cloud Control to monitor an Oracle HTTP Server(OHS) target, notifications are being raised that the target is unavailable. However using the FMW command opmnctl status, the OHS target is shown as up and no issues are seen using it. See the following examples of the errors seen.

 

From an email notificatiion on the OHS-related alert:

Host=<hostname>
Target type=Oracle HTTP Server
Target name=/OFM_Farm01_base_domain/instance1/ohs1
Categories=Error
Message=The percentage of requests that resulted in errors is 100%
Severity=Critical
Event reported time=Aug 27, 2012 12:38:33 PM EST
Associated Incident Id=15895
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=ohs_server:error.currentPercentage
Metric Group=OHS Server Metrics
Metric=Error Rate (%)
Metric value=100.0
Key Value=
Rule Name=DEFAULT_RULESET_FOR_ALL_TARGETS,METRIC_ALERT_INCIDENT_CREATION
Rule Owner=

 

From the .../agent_inst/sysman/log/gcagent.log concerning the outage:

2012-08-29 11:57:51,622 [34:B397303E] INFO - observed collection value for
column "handle.maxTime" is null for metric ohs_server on oracle_apache target
/OFM_Farm01_base_domain/instance1/ohs1; removing
2012-08-29 11:57:51,623 [34:B397303E] INFO - observed collection value for
column "internalRedirect.count" is null for metric ohs_server on
oracle_apache target /OFM_Farm01_base_domain/instance1/ohs1; removing
2012-08-29 11:57:51,624 [34:B397303E] INFO - (end processing requests for
OMS@10020@Hostname=>[134620537820007])
2012-08-29 11:57:51,639 [34:B397303E] INFO - (begin processing requests for
OMS@10020@Hostname=>[134620537850007])
2012-08-29 11:57:51,639 [34:B397303E] INFO - >>> Dispatching request:
GetMetricDataRequest
(ohs_server@oracle_apache./OFM_Farm01_base_domain/instance1/ohs1 {}) <<<
2012-08-29 11:57:51,817 [34:B397303E] INFO - >>> Reporting response:
GetMetricDataResponse
(ohs_server@oracle_apache./OFM_Farm01_base_domain/instance1/ohs1
2012-08-29T11:57:51.639+10:00 colu

agent OHS1 - Managed Targets: DOWN

 

And finally the following was also noted in the gcagent.log

2012-08-22 16:00:20,436 [76:E8B52C60:GC.Executor.6 (RecalcTargetOnSchedule
for oracle_apache./OFM_Farm01_base_domain/instance1/ohs1)] INFO -
Recalculating target [oracle_apache./OFM_Farm01_base_domain/instance1/ohs1]
dynamic properties
2012-08-22 16:01:22,320 [76:E8B52C60] INFO - Recalculating target
[oracle_apache./OFM_Farm01_base_domain/instance1/ohs1] dynamic properties
2012-08-22 16:04:24,213 [75:4139529F:GC.Executor.5 (RecalcTargetOnSchedule
for oracle_apache./OFM_Farm01_base_domain/instance1/ohs1)] INFO -
Recalculating target [oracle_apache./OFM_Farm01_base_domain/instance1/ohs1]
dynamic properties

Note: The above Recalculations occured very frequently for some time before the reported outage in Cloud Control
        These calculations are usually done only once, at start up or at set times and should not be happening frequently.
        This is a good indication of a problem.

Finally the following was also seen in the gcagent.log

2012-08-27 13:25:36,384 [149:35FC5C71] INFO - (end processing requests for
OMS@10020@Hostname=>[134603785860006])

....

2012-08-28 11:39:58,620 [149:35FC5C71] INFO - (begin processing requests for
OMS@10020@Hostname=>[134611799850001])

**: These Begin and End processing is referring to communication between the Agent and OMS. These are often there and
     by themselves do not indicate an error, but in this condition in conjunction with the Recalculation messages they can
     indicate an outage

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms