OCI Application Performance Monitoring Service : Agent Metric Collection Issues Due To APM Custom Metrics
(Doc ID 2995181.1)
Last updated on DECEMBER 26, 2023
Applies to:
OCI Application Performance Monitoring Service - Version NA and laterInformation in this document applies to any platform.
Symptoms
In Observability & Management (O&M), observed multiple issues:
1. Management Agent not loading the data.
2. Multiple resources not emitting data.
3. False alerts were reported due to the data not being loaded.
Observing 400 & 500 errors in the agent logs.
Error stack from /opt/oracle/mgmt_agent/agent_inst/log/mgmt_agent.log file:
<--> Endpoint: telemetry-ingestion.us-ashburn-1.oraclecloud.com
opc-request-id: LS6H078JNXJNMUG675V7T0HY65N811YB
StartTime: 2023-11-29 12:46:10,188 GMT
Status: 400 Bad Request
Headers: transfer-encoding=chunked
content-type=application/json
connection=keep-alive
Content-Length=-1
opc-request-id=<OPC Request ID>
Date=Wed, 29 Nov 2023 12:46:10 GMT
ErrorBody: {"code":"InvalidParameter","message":"No valid metrics. All metrics failed input validation. First failure: POST stream limit for metricGroup: MetricDataDetails(namespace=oracle_oci_database_cluster, resourceGroup=oracle_lsnr, metadata={},...(truncated)...
EndTime: 2023-11-29 12:46:10,215 GMT
opc-request-id: LS6H078JNXJNMUG675V7T0HY65N811YB
StartTime: 2023-11-29 12:46:10,188 GMT
Status: 400 Bad Request
Headers: transfer-encoding=chunked
content-type=application/json
connection=keep-alive
Content-Length=-1
opc-request-id=<OPC Request ID>
Date=Wed, 29 Nov 2023 12:46:10 GMT
ErrorBody: {"code":"InvalidParameter","message":"No valid metrics. All metrics failed input validation. First failure: POST stream limit for metricGroup: MetricDataDetails(namespace=oracle_oci_database_cluster, resourceGroup=oracle_lsnr, metadata={},...(truncated)...
EndTime: 2023-11-29 12:46:10,215 GMT
<--> Endpoint: telemetry-ingestion.us-ashburn-1.oraclecloud.com
opc-request-id: P40JLG7OVDW7VBK5C8BYNC45N0BLECQM
StartTime: 2023-12-01 20:33:20,702 GMT
Status: 500 Internal Server Error
Headers: transfer-encoding=chunked
content-type=application/json
connection=keep-alive
Content-Length=-1
opc-request-id=P40JLG7OVDW7VBK5C8BYNC45N0BLECQM/7079950C6355FB6D7ECB122541EA0CE3/523E8CAF760CCB48115FF4A06F3400FF
Date=Fri, 1 Dec 2023 20:33:20 GMT
ErrorBody: {"code":"InternalError","message":"Internal Server Error"}
EndTime: 2023-12-01 20:33:22,594 GMT2023-12-01 20:33:22,597 [SendQueue.1 (SenderManager_sender)-67] INFO - SendStatus processing exception for file: /opt/oracle/mgmt_agent/agent_inst/state/dataStorage/partition_80/680_Metric_Telemetry_host%5flinux.schpcpsdb1.schpdbpvt.schprod.oraclevcn.com.gz
2023-12-01 20:33:22,598 [SendQueue.1 (SenderManager_sender)-67] INFO - SenderStatus: HttpException exception [500] - limit enabled
opc-request-id: P40JLG7OVDW7VBK5C8BYNC45N0BLECQM
StartTime: 2023-12-01 20:33:20,702 GMT
Status: 500 Internal Server Error
Headers: transfer-encoding=chunked
content-type=application/json
connection=keep-alive
Content-Length=-1
opc-request-id=P40JLG7OVDW7VBK5C8BYNC45N0BLECQM/7079950C6355FB6D7ECB122541EA0CE3/523E8CAF760CCB48115FF4A06F3400FF
Date=Fri, 1 Dec 2023 20:33:20 GMT
ErrorBody: {"code":"InternalError","message":"Internal Server Error"}
EndTime: 2023-12-01 20:33:22,594 GMT2023-12-01 20:33:22,597 [SendQueue.1 (SenderManager_sender)-67] INFO - SendStatus processing exception for file: /opt/oracle/mgmt_agent/agent_inst/state/dataStorage/partition_80/680_Metric_Telemetry_host%5flinux.schpcpsdb1.schpdbpvt.schprod.oraclevcn.com.gz
2023-12-01 20:33:22,598 [SendQueue.1 (SenderManager_sender)-67] INFO - SenderStatus: HttpException exception [500] - limit enabled
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |