On Oracle Big Data Appliance Secure CDH Cluster HDFS Services on a Particular Node Fail to Start (Doc ID 2061178.1)

Last updated on OCTOBER 11, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.1.0 and later
Linux x86-64

Symptoms

On Oracle Big Data Appliance secure CDH cluster, many HDFS services fail to start on a particular node.

Datanode log indicates below error:

2015-09-25 09:40:13,136 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: Login failure for hdfs/<FQDNofBDANode>@<REALMName> from keytab hdfs.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6)


Also kinit using the principal from hdfs.keytab fails with the same error.

Command to list the principals from the hdfs.keytab file created by Cloudera Manager(CM):

# klist -e -t -k `ls -lrtd /var/run/cloudera-scm-agent/process/*DATA* | tail -1 | awk '{print$9}'`/hdfs.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/469-hdfs-DATANODE/hdfs.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 09/04/15 18:19:51 hdfs/<FQDNofBDANode>@<REALMName> (aes256-cts-hmac-sha1-96)
 ..........

2 09/04/15 18:19:51 hdfs/<FQDNofBDANode>@<REALMName> (arcfour-hmac)

 kinit command using the principal listed above fails with same error:

# kinit hdfs/<FQDNofBDANode>@<REALMName> -k -t `ls -lrtd /var/run/cloudera-scm-agent/process/*DATA* | tail -1 | awk '{print$9}'`/hdfs.keytab
[52552] 1443214976.851343: Getting initial credentials for hdfs/<FQDNofBDANode>@<REALMName>
[52552] 1443214976.853158: Looked up etypes in keytab: rc4-hmac
[52552] 1443214976.853201: Sending request (242 bytes) to <REALMName>
[52552] 1443214976.853247: Resolving hostname <FQDNofBDANode>
[52552] 1443214976.853588: Sending initial UDP request to dgram 10.**.**.**:88
[52552] 1443214976.892024: Received answer from dgram 10.**.***.**:88
[52552] 1443214976.892061: Response was not from master KDC
[52552] 1443214976.892129: Received error from KDC: -1765328378/Client not found in Kerberos database
[52552] 1443214976.892172: Getting initial credentials for hdfs/<FQDNofBDANode>@<REALMName>
[52552] 1443214976.892253: Looked up etypes in keytab: rc4-hmac
[52552] 1443214976.892283: Sending request (242 bytes) to <FQDNofAD> (master)
kinit: Client not found in Kerberos database while getting initial credentials

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms