My Oracle Support Banner

Client/Edge Node BDA Access Fails with "ConnectTimeoutException: 60000 millis" Due to Private IP Address Return for DataNode (Doc ID 2068582.1)

Last updated on NOVEMBER 17, 2023

Applies to:

Big Data Appliance Integrated Software - Version 4.0 and later
Linux x86-64

Symptoms

On a BDA client/edge node access to the BDA from a Java job raises:

15/10/19 11:17:55 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=/<IB& bdanode0x>:50010]
  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
  at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3106)
  at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:755)
  at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:670)
  at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:337)
  at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:621)
  at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:847)
  at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:897)
  at java.io.DataInputStream.read(DataInputStream.java:100)
  at oracle.apps.logtracer.main.IncidentTracerBDA.readAndCreateZipFile(IncidentTracerBDA.java:272)
  at oracle.apps.logtracer.main.IncidentTracerBDA.processZipFiles(IncidentTracerBDA.java:321)
  at oracle.apps.logtracer.main.IncidentTracerBDA.main(IncidentTracerBDA.java:217)
15/10/19 11:17:55 WARN hdfs.DFSClient: Failed to connect to /<IB& bdanode0x>:50010 for block, add to deadNodes
and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for
channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending
remote=/<IB& bdanode0x>:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel
to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/<IB& bdanode0x>:50010]


On the BDA both of these are set in Cloudera Manager:

Use DataNode Hostname from hdfs Service-Wide
dfs.client.use.datanode.hostname

and

Use DataNode Hostname from DataNode Default Group
dfs.datanode.use.datanode.hostname

But these settings do not seem to be honored and return the Infiniband IP address of the DataNode which is not accessible to a client/edge node.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.