Upgrade to Mammoth 4.5.0 Fails on Step 7 at Hadoop::Startcloudera/Exec[start_cloudera_services] Due to Navigator Metadata Server Failing to Start with Keytab Errors (Doc ID 2202703.1)

Last updated on NOVEMBER 15, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Information in this document applies to any platform.

Symptoms

Upgrade to Mammoth 4.5.0 fails on Step 7 with:

ERROR: Puppet agent run on node bdanode01 had errors. List of errors follows
************************************
Error [60743]: (//bdanode01.example.com//Stage[main]/Hadoop::Startcloudera/Exec[start_cloudera_services]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera_1478647016.out returned 1 instead of one of [0]
************************************

1. The file startcloudera_1478647016.out shows:

Command ID is 10355
..................
Command 10355 finished after 95 seconds
Operation failed
Result Message is: "Command 'Start' failed for cluster '<cluster name>'",

2. commands_10355.out shows:

{
"id" : 10355,
"name" : "Restart",
"startTime" : "2016-11-09T01:07:44.523Z",
"endTime" : "2016-11-09T01:09:17.103Z",
"active" : false,
"success" : false,
"resultMessage" : "Command 'Start' failed for cluster '<cluster name>'",
"clusterRef" : {
"clusterName" : "<cluster name>"
},
"children" : {
"items" : [ {
"id" : 10356,
"name" : "Stop",
"startTime" : "2016-11-09T01:07:44.777Z",
"endTime" : "2016-11-09T01:08:15.535Z",
"active" : false,
"success" : true,
"resultMessage" : "All services successfully stopped.",
"clusterRef" : {
"clusterName" : "<cluster name>"
}
}, {
"id" : 10391,
"name" : "Start",
"startTime" : "2016-11-09T01:08:15.857Z",
"endTime" : "2016-11-09T01:09:17.103Z",
"active" : false,
"success" : false,
"resultMessage" : "At least one service failed to start.",
"clusterRef" : {
"clusterName" : "<cluster name>"
}
} ]
},
"canRetry" : false
}


3. Cloudera Manager (CM) shows mgmt in bad health. The Navigator Metadata Server is down.

4. Navigating to All Recent Commands show an error in the Navigator Metadata Server regarding a bad keytab file.

5. Regenerating the missing credentials in CM via Administration > Security > Kerberos > Generate Missing Credentials raises:

ldap_add: Insufficient access (50)
additional info: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms