My Oracle Support Banner

Yarn NodeManagers Fail To Start with "Java.io.IOException: Node Mismatch After Server Started" When Name Resolution Incorrectly Resolves Hosts to Client and not Private IP Addresses (Doc ID 2334093.1)

Last updated on JULY 26, 2021

Applies to:

Big Data Appliance Integrated Software - Version 4.9.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

The Yarn NodeManagers on all cluster hosts fail to start. The associated NodeManager log at /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-bdanode0x.example.com.log.out shows:

Impl failed in state STARTED; cause: java.io.IOException: Node mismatch after server started, expected '<CLIENT_IP_ADDRESS>:8041' but found 'bdanode0x.example.com:8041'

Because of this most services in Cloudera Manager are down (and fail to start).

Additional symptoms are:

1. bdacheckcluster shows much of the cluster is down. For example on a 6 node cluster output is like:

allows for proper name resolution. However /etc/hosts on the BDA should should not be modified as described in: Details about /etc/hosts File on Oracle Big Data Appliance(Doc ID 1624234.1). Therefore the workaround is really another problem symptom.

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
 Root Cause
 Troubleshooting
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.