Oracle Big Data Appliance V2.3.1 Mammoth Install Step 16 is Slow During Cluster Health Checks: "connection timed out" Found in bdacheckcluster.log (Doc ID 1602183.1)

Last updated on MARCH 23, 2014

Applies to:

Big Data Appliance Integrated Software - Version 2.3.1 to 2.3.1 [Release 2.3]
Linux x86-64

Symptoms

Running the cluster validation checks during Step 16 of the Mammoth install seems slow.

Examining the bdacheckcluster.log file in:

/opt/oracle/bda/install/log/clusterchk/summary-<timestamp>/bdacheckcluster_<id>/bdacheckcluster.log

show that all tests are successful, however, the output for the DNS checks to verify that the public hostname for each server is available on each of the server of the cluster, starting after the INFO message:

INFO: Checking local DNS resolve of public hostnames on all cluster hosts

shows numerous messages indicating that the connection timed out:

;; connection timed out; no servers could be reached


This step is verifying the DNS resolution of each server hostname from each server.  In the  case of a full rack, for example, there will be 18 X 18 checks as each server verifies the name resolution for each server.  Therefore, timed out connections can slow down this phase of the health checks.

Sample output in bdacheckcluster.log looks like:

INFO: Starting Big Data Appliance Cluster health checks at Wed Nov 20 20:21:57 2013
0
SUCCESS: Mammoth configuration file is valid.
INFO: Checking health of Cloudera Manager service
SUCCESS: hdfs is in good health
...
SUCCESS: Cluster passed checks on all hadoop services health check
INFO: Checking health of All Hosts
SUCCESS: bdanode01 is in good health
...
INFO: Checking local DNS resolve of public hostnames on all cluster hosts

bdanode01 has address *.*.*.1
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode01 to *.*.*.1 on bdanode01

bdanode02 has address *.*.*.2
SUCCESS: Correct result resolving bdanode02 to *.*.*. 2 on bdanode01

bdanode03 has address *.*.*.3
SUCCESS: Correct result resolving bdanode03 to *.*.*.3 on bdanode01
...
bdanode0n has address *.*.*.n
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode0n to *.*.*.n on bdanode01

------------------------------

bdanode02 has address *.*.*.2
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode02 to *.*.*.2 on bdanode02

bdanode03 has address *.*.*.3
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode03 to *.*.*.3 on bdanode02
...
bdanode0n has address *.*.*.n
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode0n to *.*.*.n on bdanode02

bdanode01 has address *.*.*.1
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode01 to *.*.*.1 on bdanode02

...
bdanode0n has address *.*.*.n
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode0n to *.*.*.n on bdanode0n

bdanode01 has address *.*.*.1
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode01 to *.*.*.1 on bdanode0n

...
bdanode0n-1 has address *.*.*.n-1
;; connection timed out; no servers could be reached
SUCCESS: Correct result resolving bdanode0n-1 to *.*.*.n-1 on bdanode0n
....
<continue on with validations>

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms