Very Slow Hive and Impala Query Performance and Errors in Logs like: "...java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.ch ..." (Doc ID 2519800.1)

Last updated on APRIL 17, 2023

Applies to:

Symptoms

A BDA cluster exhibits increased query times and slow performance when running hive and Impala jobs. In this cluster, users typically access both applications via the web UI in Oozie and hue, but slow performance is also seen with the client applications.

For example, some jobs that normally take 5 minutes are taking more than one hour. Other jobs are taking five times longer and hanging the entire cluster which then requires cluster restart to recover.

There are no errors displayed in the web UI, but there are intermittent errors in the Impala daemon (Impalad) logs like:

"W0302 17:00:15.843978 48829 DFSInputStream.java:865] Exception while reading from BP-569733022-<LOCAL_IP>:<BLOCK_ID> of /<USER>/<USERNAME>/<DIR>/<LOGNAME>.parq from DatanodeInfoWithStorage ...java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/<IP>:<PORT> remote=<DATANODE_FQ_HOSTNAME>/<REMOTE_IP>:<PORT>]"

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.