Client/Edge Node BDA Access Fails with "ConnectTimeoutException: 60000 millis" Due to Private IP Address Return for DataNode (Doc ID 2068582.1)

Last updated on OCTOBER 20, 2015

Applies to:

Big Data Appliance Integrated Software - Version 4.0 and later
Linux x86-64

Symptoms

On a BDA client/edge node access to the BDA from a Java job raises:

15/10/19 11:17:55 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=/<IB& bdanode0x>:50010]
  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
  at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3106)
  at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:755)
  at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:670)
  at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:337)
  at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:621)
  at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:847)
  at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:897)
  at java.io.DataInputStream.read(DataInputStream.java:100)
  at oracle.apps.logtracer.main.IncidentTracerBDA.readAndCreateZipFile(IncidentTracerBDA.java:272)
  at oracle.apps.logtracer.main.IncidentTracerBDA.processZipFiles(IncidentTracerBDA.java:321)
  at oracle.apps.logtracer.main.IncidentTracerBDA.main(IncidentTracerBDA.java:217)
15/10/19 11:17:55 WARN hdfs.DFSClient: Failed to connect to /<IB& bdanode0x>:50010 for block, add to deadNodes
and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for
channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending
remote=/<IB& bdanode0x>:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel
to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/<IB& bdanode0x>:50010]


On the BDA both of these are set in Cloudera Manager:

Use DataNode Hostname from hdfs Service-Wide
dfs.client.use.datanode.hostname

and

Use DataNode Hostname from DataNode Default Group
dfs.datanode.use.datanode.hostname

But these settings do not seem to be honored and return the Infiniband IP address of the DataNode which is not accessible to a client/edge node.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms