Flume Agents Do Not Read from the Beginning Offset of a Kafka Source

(Doc ID 2153775.1)

Last updated on JUNE 16, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.3.0 and later
Linux x86-64

Symptoms

With Flume agent configuration file for the Source as below, Flume agents do not read from the beginning offset of a Kafka source 

# Source(s):
agent_phx_audit.sources.src.selector,type=replicating
agent_phx_audit.sources.src.channels=ch ch1 ch2 agent_phx_audit.sources.src.type=org.apache.flume.source.kafka.KafkaSource
agent_phx_audit.sources.src.zookeeperConnect=bda1node01.example.com:2181,bda1node02.example.com:2181,bda1node03.example.com:2181
agent_phx_audit.sources.src.topic=phx-audit
agent_phx_audit.sources.src.batchSize=1024
agent_phx_audit.sources.src.parseAsFlumeEvent=false
agent_phx_audit.sources.src.readSmallestOffset=true

Messags are being read from the the previous day whereas the output of the kafka-console-consumer provides an output for messages starting from two weeks before. 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms