Enabling MIT Kerberos with "bdacli enable kerberos" Fails at Hadoop::Startclouderakerberos Due to "mgmt" Service Failure to Start with "Role is missing Kerberos keytab" (Doc ID 2188314.1)

Last updated on SEPTEMBER 29, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

The symptoms are:

1. Enabling MIT Kerberos with "bdacli enable kerberos"  fails as below:

************************************
Error [29808]: (//bdanode01.example.com//Stage[main]/Hadoop::Startclouderakerberos/Exec[start_cloudera_services_kerberos]/returns) change from notrun to 0 failed:/opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos_<id>.out returned 1 instead of one of [0]
************************************

2. /opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos_<id>.out reports "Failed to restart service." on command 8000.

Output looks like:

Command 8000 finished after 50 seconds
Operation failed
Result Message is: "Failed to restart service."

3. commands_8000.out shows the mgmt service will not start.

Output looks like:

...
"id" : 8000,
"name" : "Restart",
"startTime" : "2016-09-27T20:24:57.113Z",
"endTime" : "2016-09-27T20:25:42.948Z",
"active" : false,
"success" : false,
"resultMessage" : "Failed to restart service.",
"serviceRef" : {
"serviceName" : "mgmt"
...

4. Checking "All Recent Commands" in Cloudera Manager (CM) shows the mgmt service will not start because the below management services fail to start with "Role is missing Kerberos keytab":

a) CM > All Recent Commands shows:

Reports Manager (bdanode 3) Role is missing Kerberos keytab
Service Monitor (bdanode 3) Role is missing Kerberos keytab
Activity Monitor (bdanode 3) Role is missing Kerberos keytab
Navigator Metadata Server (bdanode 3) Role is missing Kerberos keytab

b) Drilling down into each of the above services in "All Recent Commands" shows the same error:

Command failed to run because this role has invalid configuration.
Review and correct its configuration. First error: Role is missing Kerberos keytab.

5. Checking on generating the credentials in CM shows that no credentials have been set up yet.

6. CM > All Recent Commands shows the "Generate Missing Credentials Command" which looks like:

Generate Missing Credentials Command
Summary
Status: Failed Start Time: Sep 27, 1:19:31 PM Duration: 5.16s
/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=EXAMPLE.COM
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf12346539834782.keytab
+ PRINC=HTTP/bdanode01.example.com@EXAMPLE.COM
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@EXAMPLE.COM -r EXAMPLE.COM'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb52587610385590192190.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb38192374134170.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb38192374134170.conf
+ kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@EXAMPLE.COM -r EXAMPLE.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey HTTP/bdanode01.example.com@EXAMPLE.COM'
kadmin: Improper format of Kerberos configuration file while initializing krb5 library

7. From the "Generate Missing Credentials" command see:

kadmin: Improper format of Kerberos configuration file while initializing krb5 library

This indicates something is wrong with the /etc/krb5.conf file. However in this case review of the /etc/krb5.conf file on all cluster nodes indicates that the file is correct.

8. From the "Generate Missing Credentials" command also see:

+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb38192374134170.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb38192374134170.conf
+ kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@EXAMPLE.COM -r EXAMPLE.COM -q 'addprinc -maxrenewlife "432000 sec" -randkey HTTP/bdanode01.example.com@EXAMPLE.COM'
kadmin: Improper format of Kerberos configuration file while initializing krb5 library

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms