My Oracle Support Banner

AVDF DBFW Resilient Pair Failed Over Due To Missed Heartbeat When rsyslogd Was Stalled (Doc ID 2317851.1)

Last updated on MARCH 26, 2019

Applies to:

Oracle Audit Vault and Database Firewall - Version 12.1.2.6 and later
Information in this document applies to any platform.

Symptoms

NOTE: In the images and/or the document content below, the user information and data used represents fictitious data from the Oracle sample schema(s) or Public Documentation delivered with an Oracle database product. Any similarity to actual persons, living or dead, is purely coincidental and not intended in any manner. 

Resilient DBFW pair failed over due to missed heartbeat. However, there was no obvious
problems or errors in DBFW processes or network why heartbeat did not reach AVS.

/var/log/messages on AVS showed there was no heartbeat received from
the enforcement point (EP_TEST) on DBFW server fw01 for 30 seconds
and considered to be lost.

<DATE> 04:02:01 avs01 rsyslogd: [origin software="rsyslogd" swVersion="3.2
2.1" x-pid="6526" x-info="http://www.rsyslog.com"] (re)start
<DATE> 04:02:05 avs01 mwecsvc: com.oracle.dbfw.MwecSyslogMessages WARN -
Enforcement point missed a heartbeat (4 seconds since last heartbeat)
(EP_TEST@fw01)
<DATE> 04:02:05 avs01 mwecsvc: com.oracle.dbfw.MwecSyslogMessages WARN -
Enforcement point missed a heartbeat (4 seconds since last heartbeat)
(EP_TEST@fw01)
<DATE> 04:02:32 avs01 mwecsvc: com.oracle.dbfw.MwecSyslogMessages WARN -
Enforcement point lost, missed 30 consecutive heartbeats (EP_TEST@fw01)
<DATE> 04:02:32 avs01 mwecsvc: com.oracle.dbfw.MwecSyslogMessages WARN -
Enforcement point lost, missed 30 consecutive heartbeats (EP_TEST@fw01)
<DATE> 04:02:33 avs01 mwecsvc: com.oracle.dbfw.MwecSyslogMessages WARN -
Firewall lost, no heartbeats for 30 seconds (fw01)

Looking at /var/log/messages on primary DBFW, rsyslogd restarted everyday at 4:02
in this system.

<DATE> 04:02:02 fw01 kernel: imklog 3.22.1, log source = /proc/kmsg started.
<DATE> 04:02:02 fw01 rsyslogd: [origin software="rsyslogd"
swVersion="3.22.1" x-pid="8779" x-info="http://www.rsyslog.com"] (re)start

/var/log/messages on Primary FW shows no error but rsyslogd was restarted at
4:03 which is the same time this issue occurred. So there was some delay restarting rsyslogd.

<DATE> 04:03:02 fw01 kernel: imklog 3.22.1, log source = /proc/kmsg started.
<DATE> 04:03:02 fw01 rsyslogd: [origin software="rsyslogd"
swVersion="3.22.1" x-pid="8779" x-info="http://www.rsyslog.com"] (re)start

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.