My Oracle Support Banner

Upgrade to Mammoth 4.5.0 Fails on Step 7 at Hadoop::Startcloudera/Exec[start_cloudera_services] Due to Navigator Metadata Server Failing to Start with Keytab Errors (Doc ID 2202703.1)

Last updated on MARCH 22, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Information in this document applies to any platform.

Symptoms

NOTE: In the images, examples and document that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

Upgrade to Mammoth 4.5.0 fails on Step 7 with:

ERROR: Puppet agent run on node bdanode01 had errors. List of errors follows
************************************
Error [60743]: (//bdanode01.example.com//Stage[main]/Hadoop::Startcloudera/Exec[start_cloudera_services]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera_<EPOCH_TIMESTAMP>.out returned 1 instead of one of [0]
************************************

1. The file startcloudera_<EPOCH_TIMESTAMP>.out shows:

Command ID is 10355
..................
Command 10355 finished after 95 seconds
Operation failed
Result Message is: "Command 'Start' failed for cluster '<CLUSTER_NAME>'",

2. commands_10355.out shows:

{
"id" : 10355,
"name" : "Restart",
"startTime" : "2016-11-09T01:07:44.523Z",
"endTime" : "2016-11-09T01:09:17.103Z",
"active" : false,
"success" : false,
"resultMessage" : "Command 'Start' failed for cluster '<CLUSTER_NAME>'",
"clusterRef" : {
"clusterName" : "<CLUSTER_NAME>"
},
"children" : {
"items" : [ {
"id" : 10356,
"name" : "Stop",
"startTime" : "2016-11-09T01:07:44.777Z",
"endTime" : "2016-11-09T01:08:15.535Z",
"active" : false,
"success" : true,
"resultMessage" : "All services successfully stopped.",
"clusterRef" : {
"clusterName" : "<CLUSTER_NAME>"
}
}, {
"id" : 10391,
"name" : "Start",
"startTime" : "2016-11-09T01:08:15.857Z",
"endTime" : "2016-11-09T01:09:17.103Z",
"active" : false,
"success" : false,
"resultMessage" : "At least one service failed to start.",
"clusterRef" : {
"clusterName" : "<CLUSTER_NAME>"
}
} ]
},
"canRetry" : false
}


3. Cloudera Manager (CM) shows mgmt in bad health. The Navigator Metadata Server is down.

4. Navigating to All Recent Commands show an error in the Navigator Metadata Server regarding a bad keytab file.

5. Regenerating the missing credentials in CM via Administration > Security > Kerberos > Generate Missing Credentials raises:

ldap_add: Insufficient access (50)
additional info: 00000005: SecErr: DSID-<DSID>, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.