My Oracle Support Banner

On Oracle Big Data Appliance Secure CDH Cluster HDFS Services on a Particular Node Fail to Start (Doc ID 2061178.1)

Last updated on JUNE 12, 2021

Applies to:

Big Data Appliance Integrated Software - Version 4.1.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

On Oracle Big Data Appliance secure CDH cluster, many HDFS services fail to start on a particular node.

Datanode log indicates below error:

2015-09-25 09:40:13,136 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: Login failure for hdfs/<FQDNofBDANode>@<REALM> from keytab hdfs.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6)


Also kinit using the principal from hdfs.keytab fails with the same error.

Command to list the principals from the hdfs.keytab file created by Cloudera Manager(CM):

# klist -e -t -k `ls -lrtd /var/run/cloudera-scm-agent/process/*DATA* | tail -1 | awk '{print$9}'`/hdfs.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/469-hdfs-DATANODE/hdfs.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 09/04/15 18:19:51 hdfs/<FQDNofBDANode>@<REALM> (aes256-cts-hmac-sha1-96)
 ..........

2 09/04/15 18:19:51 hdfs/<FQDNofBDANode>@<REALM> (arcfour-hmac)

 kinit command using the principal listed above fails with same error:

# kinit hdfs/<FQDNofBDANode>@<REALM> -k -t `ls -lrtd /var/run/cloudera-scm-agent/process/*DATA* | tail -1 | awk '{print$9}'`/hdfs.keytab
[52552] 1443214976.851343: Getting initial credentials for hdfs/<FQDNofBDANode>@<REALM>
[52552] 1443214976.853158: Looked up etypes in keytab: rc4-hmac
[52552] 1443214976.853201: Sending request (242 bytes) to <REALM>
[52552] 1443214976.853247: Resolving hostname <FQDNofBDANode>
[52552] 1443214976.853588: Sending initial UDP request to dgram <IP_ADDRESS>:88
[52552] 1443214976.892024: Received answer from dgram <IP_ADDRESS>:88
[52552] 1443214976.892061: Response was not from master KDC
[52552] 1443214976.892129: Received error from KDC: -1765328378/Client not found in Kerberos database
[52552] 1443214976.892172: Getting initial credentials for hdfs/<FQDNofBDANode>@<REALM>
[52552] 1443214976.892253: Looked up etypes in keytab: rc4-hmac
[52552] 1443214976.892283: Sending request (242 bytes) to <FQDNofAD> (master)
kinit: Client not found in Kerberos database while getting initial credentials

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.