During a BDA 4.6 CDH 5.8.0 to 5.8.2 Rolling Upgrade Kafka Service Fails to Restart (Doc ID 2196113.1)

Last updated on OCTOBER 22, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.6.0 and later
Linux x86-64

Symptoms

The BDA V4.6 CDH 5.8.0 to 5.8.0 rolling upgrade fails to restart the Kafka service. Step 16 of 19 fails with:

Failed to upgrade cluster
Completed 16 of 19 step(s).
Failed to perform Rolling Restart on the cluster.
Stop this Kafka Broker >>> Role must be started.

 

1. The Kafka Brokers fail to start.  The Role Log shows "kafka.common.InconsistentBrokerIdException":

Oct 21, 5:22:06.685 PM FATAL kafka.server.KafkaServer
Fatal error during KafkaServer startup. Prepare to shutdown
kafka.common.InconsistentBrokerIdException: Configured broker.id 111 doesn't match stored broker.id 61 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:635)
at kafka.server.KafkaServer.startup(KafkaServer.scala:184)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)

 

2. There is an additional failure when restarting the Kafka service where the Kafka Mirror Maker fails to start.  The stderr log shows:

Exception in thread "main" java.lang.NullPointerException
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:291)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)

The Role Log shows:

ERROR kafka.tools.MirrorMaker$
Exception when starting mirror maker.
org.apache.kafka.common.config.ConfigException: Invalid value smallest for configuration auto.offset.reset: String must be one of: latest, earliest, none

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms