After BDA Non-Critical Node Reprovision the Host DataNode Fails to Start with "This host has been out of contact with Cloudera Manager for too long" (Doc ID 2235397.1)

Last updated on FEBRUARY 21, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

After BDA node reprovision on a non-critical node, the node fails to start with the following errors:

1. In Cloudera Manager (CM) navigating: Hosts shows the host is not heart-beating into the cluster.

There is nothing in the "Last Heartbeat" field. This is the case even though the CM agent is running on the node, i.e. "service cloudera-scm-agent status" shows it is up.

2. In CM navigating to Hosts > <reprovisioned node>, Show a "Heath Test" failure of

This host has been out of contact with Cloudera Manager for too long.
The host's Cloudera Manager Agent's software version can not be determined.

3. In CM navigating Hosts > Parcels, show no parcel distributed to the host. The error reports:

No parcel based processes running on this host

Other symptoms in  Hosts > Parcels are:

a) This error is reported existing parcels:

Local parcel error for parcel <parcel_name>-unknown.
One or more hosts did not report their OS Distribution.
This is likely due to running an outdated version of the Cloudera Manager Host Inspector to check the Agent versions.

b) The CDH parcel reports:

Local parcel error for parcel <CDH parcel>-unknown
One or more hosts did not report their OS Distribution.
This is likely due to running an outdated version of the Cloudera Manager Host Inspector to check the Agent versions.
Error when distributing to bdanode0x.example.com: Host is in bad health.

c) /opt/cloudera/parcels is empty on the the reprovisioned node.

4. On the reprovisioned host:

a) /var/log shows no DataNode logs are generated.

b) The hdfs/yarn users are missing on the reprovisioned host.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms