Enabling MIT Kerberos with "bdacli enable kerberos" Fails at Hadoop::Startclouderakerberos Due to "mgmt" Service Failure to Start with "Role is missing Kerberos keytab"
(Doc ID 2188314.1)
Last updated on APRIL 13, 2022
Applies to:
Big Data Appliance Integrated Software - Version 4.5.0 and laterLinux x86-64
Symptoms
The symptoms are:
1. Enabling MIT Kerberos with "bdacli enable kerberos" fails as below:
Error [29808]: (//bdanode01.example.com//Stage[main]/Hadoop::Startclouderakerberos/Exec[start_cloudera_services_kerberos]/returns) change from notrun to 0 failed:/opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos_<ID>.out returned 1 instead of one of [0]
************************************
2. /opt/oracle/BDAMammoth/bdaconfig/tmp/startclouderakerberos_<ID>.out reports "Failed to restart service." on command 8000.
Output looks like:
Operation failed
Result Message is: "Failed to restart service."
3. commands_8000.out shows the mgmt service will not start.
Output looks like:
"id" : 8000,
"name" : "Restart",
"startTime" : "2016-09-27T20:24:57.113Z",
"endTime" : "2016-09-27T20:25:42.948Z",
"active" : false,
"success" : false,
"resultMessage" : "Failed to restart service.",
"serviceRef" : {
"serviceName" : "mgmt"
...
4. Checking "All Recent Commands" in Cloudera Manager (CM) shows the mgmt service will not start because the below management services fail to start with "Role is missing Kerberos keytab":
a) CM > All Recent Commands shows:
Service Monitor (bdanode 3) Role is missing Kerberos keytab
Activity Monitor (bdanode 3) Role is missing Kerberos keytab
Navigator Metadata Server (bdanode 3) Role is missing Kerberos keytab
b) Drilling down into each of the above services in "All Recent Commands" shows the same error:
Review and correct its configuration. First error: Role is missing Kerberos keytab.
5. Checking on generating the credentials in CM shows that no credentials have been set up yet.
6. CM > All Recent Commands shows the "Generate Missing Credentials Command" which looks like:
Summary
Status: Failed Start Time: Sep 27, 1:19:31 PM Duration: 5.16s
/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=<REALM>
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf<ID>.keytab
+ PRINC=HTTP/bdanode01.example.com@<REALM>
+ MAX_RENEW_LIFE=432000
+ KADMIN='kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@<REALM> -r <REALM>'
+ RENEW_ARG=
+ '[' 432000 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "432000 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb<ID>.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb<ID>.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb<ID>.conf
+ kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@<REALM> -r <REALM> -q 'addprinc -maxrenewlife "432000 sec" -randkey HTTP/bdanode01.example.com@<REALM>'
kadmin: Improper format of Kerberos configuration file while initializing krb5 library
7. From the "Generate Missing Credentials" command see:
This indicates something is wrong with the /etc/krb5.conf file. However in this case review of the /etc/krb5.conf file on all cluster nodes indicates that the file is correct.
8. From the "Generate Missing Credentials" command also see:
+ cat /var/run/cloudera-scm-server/krb<ID>.conf
+ kadmin -k -t /etc/cloudera-scm-server/cmf.keytab -p cloudera-scm/admin@<REALM> -r <REALM> -q 'addprinc -maxrenewlife "432000 sec" -randkey HTTP/bdanode01.example.com@<REALM>'
kadmin: Improper format of Kerberos configuration file while initializing krb5 library
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |