My Oracle Support Banner

Flume Agents Do Not Read from the Beginning Offset of a Kafka Source (Doc ID 2153775.1)

Last updated on JANUARY 28, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.3.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

With Flume agent configuration file for the Source as below, Flume agents do not read from the beginning offset of a Kafka source 

# Source(s):
agent_phx_audit.sources.src.selector,type=replicating
agent_phx_audit.sources.src.channels=ch ch1 ch2 agent_phx_audit.sources.src.type=org.apache.flume.source.kafka.KafkaSource
agent_phx_audit.sources.src.zookeeperConnect=bdanode01.example.com:2181,bdanode02.example.com:2181,bdanode03.example.com:2181
agent_phx_audit.sources.src.topic=phx-audit
agent_phx_audit.sources.src.batchSize=1024
agent_phx_audit.sources.src.parseAsFlumeEvent=false
agent_phx_audit.sources.src.readSmallestOffset=true

Messags are being read from the the previous day whereas the output of the kafka-console-consumer provides an output for messages starting from two weeks before. 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.