The Cloudera Navigator Metadata Service Intermittently Falls on BDA V4.4/CDH 5.5.1 with Out of Memory Errors (Doc ID 2139862.1)

Last updated on MAY 20, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 and later
Linux x86-64

Symptoms

On BDA V4.4/CDH 5.5.1 MGMT_NAVIGATORMETASERVER_HEALTH goes into "bad" health.

The stack shows:

12:50:10.844 WARN org.apache.jasper.runtime.TldScanner

... the supplied resource file:/usr/share/java/oracle-connector-java.jar does not exist
java.io.FileNotFoundException: /usr/share/java/oracle-connector-java.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:220)
at java.util.zip.ZipFile.<init>(ZipFile.java:150)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:103)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
at org.apache.jasper.runtime.TldScanner.scanJar(TldScanner.java:438)
at org.apache.jasper.runtime.TldScanner.scanJars(TldScanner.java:689)
at org.apache.jasper.runtime.TldScanner.scanTlds(TldScanner.java:350)
at org.apache.jasper.runtime.TldScanner.onStartup(TldScanner.java:239)
at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:106)
at org.eclipse.jetty.annotations.ServletContainerInitializerListener.doStart(ServletContainerInitializerListener.java:107)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.util.component.AggregateLifeCycle.doStart(AggregateLifeCycle.java:81)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:58)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:96)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:115)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1242)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:494)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:172)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95)
at org.eclipse.jetty.server.Server.doStart(Server.java:282)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at com.cloudera.nav.server.NavServer.run(NavServer.java:437)
at com.cloudera.nav.server.NavServer.main(NavServer.java:98)

 

2. The role logs show problems during the internal solr core startup.

2016-05-04 09:47:11,280 WARN org.apache.solr.update.UpdateLog: Starting log replay tlog{file=/var/lib/cloudera-scm-navigator/solr/*/data/*/log.0000000000000064928 refcount=2} active=false starting pos=0
2016-05-04 09:47:11,602 WARN org.apache.solr.update.UpdateLog: Starting log replay tlog{file=/var/lib/cloudera-scm-navigator/solr/*/data/*/log.0000000000000023308 refcount=2} active=false starting pos=0
2016-05-04 09:47:32,913 INFO com.cloudera.nav.server.NavServer: Navigator Metadata Server listening on http://<private IP navigator metadatat host>:7187
2016-05-04 09:47:32,916 ERROR org.apache.solr.update.UpdateLog: java.io.EOFException
2016-05-04 09:47:37,466 ERROR org.apache.solr.update.UpdateLog: java.io.EOFException

3. The associated stdout/stderr logs show low heap configuration resulting in OutOfMemoryException.

# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="/usr/lib64/cmf/service/common/killparent.sh"
# Executing /bin/sh -c "/usr/lib64/cmf/service/common/killparent.sh"...
Tue May 10 08:27:34 CEST 2016

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms