Step 7 of Mammoth Upgrade Fails With "Navigator Metadata Server fails to start"

(Doc ID 2308119.1)

Last updated on SEPTEMBER 30, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

Step 7 of Mammoth upgrade fails with output like:

Error [9179]: (//bdanode03.example.com//State[main]/Hadoop::Upgradeactions/Exec[set_cluster_conf]/returns)
change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.sh &>
/opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out returned 1 instead of one of [0]

Additional Symptoms are:

1. /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out points to commands_51933.out which shows output like:

a) /opt/oracle/BDAMammoth/bdaconfig/tmp/setclusterconf.out shows output like:

Command 51933 finished after 5 seconds
Operation failed
Result Message is: "Service must have either started or stopped roles.",

b) commands_51933.out shows output like:

"id" : 51933,
"name" : "Restart",
"startTime" : "2017-09-09T22:42:54.876Z",
"endTime" : "2017-09-09T22:42:54.876Z",
"active" : false,
"success" : false,
"resultMessage" : "Service must have either started or stopped roles.",
"serviceRef" : {
"serviceName" : "mgmt"
},
"children" : {
"items" : [ ]
},
"canRetry" : false
}

2. Checking Cloudera Manager (CM) shows that the Management Service, mgmt, is not starting because the Navigator Metadata Server fails to start.

3. Navigating: mgmt > Navigator Metadata Server > Instances > Process, shows:

This role's process is starting. This role is supposed to be started.

4. The Navigator Metadata Server log, /var/log/cloudera-scm-navigator/mgmt-cmf-mgmt-NAVIGATORMETASERVER-bdanode03.example.com.log.out, on the node with Navigator Metadata Server role, Node 3 by default,  shows output like:

Exception: Cursor functionality requires a sort containing a uniqueKey field tie breaker
at org.apache.solr.search.CursorMark.(CursorMark.java:104)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
...
2017-09-09 19:40:48,181 ERROR com.cloudera.nav.server.NavServer [main]: Error upgrading solr configuration or data.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96)
at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76)
at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:242)
at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:217)
at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:70)
at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:154)
at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:357)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms