Oracle Big Data Appliance V2.2/CDH 4.3 No Active NameNode After Disruption of ZooKeeper Service
Last updated on NOVEMBER 18, 2016
Applies to:Big Data Appliance Integrated Software - Version 2.2.0 and later
If there has been a disruption to the ZooKeeper service, possibly deleting ZooKeeper data under /var/log/zookeeper, Cloudera Manger (CM) may report "Bad Health" for the hdfs and/or mapreduce service. Further investigation in CM shows that both NameNodes are in Standby mode.
Services > All Services > on hdfs line: 2 NameNodes
Both NameNodes in Standby Mode:
Trying to stop /start one of the Standby NameNodes to make it Active does not work. Messaging like below is raised:
Try forcing a failover if you want the selected NameNode to be transitioned to active mode even if it is in safe mode. Forcing the failover can also be used to override fencing, after confirming that the other NameNode is not running.
Forcing a failover, however, does not transition the NameNode into an Active state.
StandbyExceptions are also raised in the NameNode logs as below. To look for such exceptions in the NameNodes logs from Cloudera Manger:
Services > All Services > on hdfs line: 2 NameNodes > Instances > Click on one namenode > Processes > Show Recent Logs
Operation category READ is not supported in state standby
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category JOURNAL is not supported in state standby
Sign In with your My Oracle Support account
Don't have a My Oracle Support account? Click to get started
My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms