My Oracle Support Banner

Step 7 of Mammoth Upgrade Fails With "Navigator Metadata Server fails to start" (Doc ID 2308119.1)

Last updated on MARCH 22, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

NOTE: In the images, examples and document that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

Step 7 of Mammoth upgrade fails with output like:

Error [9179]: (//bdanode03.example.com//State[main]/Hadoop::Upgradeactions/Exec[set_cluster_conf]/returns)
change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.sh &>
/opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out returned 1 instead of one of [0]

Additional Symptoms are:

1. /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out points to commands_51933.out which shows output like:

a) /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out shows output like:

Command 51933 finished after 5 seconds
Operation failed
Result Message is: "Service must have either started or stopped roles.",

b) commands_51933.out shows output like:

"id" : 51933,
"name" : "Restart",
"startTime" : "2017-09-09T22:42:54.876Z",
"endTime" : "2017-09-09T22:42:54.876Z",
"active" : false,
"success" : false,
"resultMessage" : "Service must have either started or stopped roles.",
"serviceRef" : {
"serviceName" : "mgmt"
},
"children" : {
"items" : [ ]
},
"canRetry" : false
}

2. Checking Cloudera Manager (CM) shows that the Management Service, mgmt, is not starting because the Navigator Metadata Server fails to start.

3. Navigating: mgmt > Navigator Metadata Server > Instances > Process, shows:

This role's process is starting. This role is supposed to be started.

4. The Navigator Metadata Server log, /var/log/cloudera-scm-navigator/mgmt-cmf-mgmt-NAVIGATORMETASERVER-bdanode03.example.com.log.out, on the node with Navigator Metadata Server role, Node 3 by default,  shows output like:

Exception: Cursor functionality requires a sort containing a uniqueKey field tie breaker
at org.apache.solr.search.CursorMark.(CursorMark.java:104)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
...
2017-09-09 19:40:48,181 ERROR com.cloudera.nav.server.NavServer [main]: Error upgrading solr configuration or data.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96)
at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76)
at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:242)
at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:217)
at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:70)
at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:154)
at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:357)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.