My Oracle Support Banner

EM13c : OMS Restarted Due To - Heartbeat Recorder Timed Out (Doc ID 2721148.1)

Last updated on OCTOBER 19, 2020

Applies to:

Enterprise Manager Base Platform - Version 13.3.0.0.0 and later
Information in this document applies to any platform.

Symptoms

Enterprise Manager (EM) Cloud Control, crashed and got restarted automatically due to "Heartbeat recorder timed out".

Error stack from <EM_INSTANCE_HOME>/em/EMGC_OMS*/sysman/log/emctl.msg file:

HealthMonitor Oct 9, 2020 3:02:26 AM OMS Heartbeat Recorder error: Heartbeat recorder timed out. OMS id=269
Critical error err=3 detected in module OMS Heartbeat Recorder
OMS will be restarted. A full thread dump will be generated in the log file
<EM_INSTANCE_HOME>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out to help Oracle Support analyse the problem.

Error stack from DB alert log file:

*************************************************************
Unable to allocate flashback log of 17040 blocks from current recovery area of size 8405385216 bytes.
Recovery Writer (RVWR) is stuck until more space is available in the recovery area.
Unable to write Flashback database log data because the recovery area is full, presence of a guaranteed
restore point and no reusable flashback logs.

Unable to write Flashback database log data because the recovery area is full, presence of a guaranteed
restore point and no reusable flashback logs.
***********************************************************************

Fatal NI connect error 12170.

VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 09-OCT-2020 12:30:25
Tracing not turned on.
Tns error struct:
ns main err code: 12535

TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505

TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505

TNS-00505: Operation timed out
nt secondary err code: 110
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.110.103.161)(PORT=35892))
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.110.103.159)(PORT=44612))

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.