Yarn NodeManagers Fail To Start with "Java.io.IOException: Node Mismatch After Server Started" When Name Resolution Incorrectly Resolves Hosts to Client and not Private IP Addresses (Doc ID 2334093.1)

(Doc ID 2334093.1)

Last updated on DECEMBER 11, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.9.0 and later
Linux x86-64

Symptoms

The Yarn NodeManagers on all cluster hosts fail to start. The associated NodeManager log at /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-bdanode0x.example.com.log.out shows:

Impl failed in state STARTED; cause: java.io.IOException: Node mismatch after server started, expected '<client IP address>:8041' but found 'bdanode0x.example.com:8041'

Because of this most services in Cloudera Manager are down (and fail to start).

Additional symptoms are:

1. bdacheckcluster shows much of the cluster is down. For example on a 6 node cluster output is like:

allows for proper name resolution. However /etc/hosts on the BDA should should not be modified as described in: Details about /etc/hosts File on Oracle Big Data Appliance(Doc ID 1624234.1). Therefore the workaround is really another problem symptom.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms