DataNode And ZooKeeper Conflict Seen On Port 50020 on Oracle Big Data Appliance V2.2 (Doc ID 1588719.1)

Last updated on MARCH 24, 2017

Applies to:

Big Data Appliance Integrated Software - Version 2.2.1 to 2.2.1 [Release 2.2]
Linux x86-64

Symptoms

Stopping Node01 of an Oracle Big Data Appliance (BDA) cluster and restarting it after some amount of time like 30 minutes shows that all the CDH services start correctly except for the DataNode. Analyzing the logs shows that Zookeeper is listening on port 50020, and the DataNode tries to listen on that port as well. Port 50020 is the designated CDH port for the DataNode i.e. it is the one a DataNode runs on.

2013-09-25 12:26:28,182 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.BindException: Problem binding to [0.0.0.0:50020] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
      at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:718)
      at org.apache.hadoop.ipc.Server.bind(Server.java:364)
      at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:462)
      at org.apache.hadoop.ipc.Server.<init>(Server.java:1841)
      at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:970)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:375)
      at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:350)
      at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695)
      at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:448)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:743)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:341)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1786)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1719)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1742)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1895)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1916)
2013-09-25 12:26:28,192 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-09-25 12:26:28,194 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at bda1node01.example.com/10.xxx.xx.xxx
************************************************************/

The lsof command shows ZooKeeper listening on port 50020:

# lsof -i:50020
COMMAND   PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    10938 zookeeper   15u  IPv4  87281      0t0  TCP *:50020 (LISTEN)
java    10938 zookeeper   38u  IPv4 140171      0t0  TCP bda1node01.example.com:50020->bda1node03.example.com:30869 (ESTABLISHED)


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms