My Oracle Support Banner

The Cloudera Navigator Metadata Service Intermittently Falls on BDA V4.4/CDH 5.5.1 with Out of Memory Errors (Doc ID 2139862.1)

Last updated on JANUARY 13, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

On BDA V4.4/CDH 5.5.1 MGMT_NAVIGATORMETASERVER_HEALTH goes into "bad" health.

The stack shows:

12:50:10.844 WARN org.apache.jasper.runtime.TldScanner

... the supplied resource file:/usr/share/java/oracle-connector-java.jar does not exist
java.io.FileNotFoundException: /usr/share/java/oracle-connector-java.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:220)
at java.util.zip.ZipFile.<init>(ZipFile.java:150)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:103)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
at org.apache.jasper.runtime.TldScanner.scanJar(TldScanner.java:438)
at org.apache.jasper.runtime.TldScanner.scanJars(TldScanner.java:689)
at org.apache.jasper.runtime.TldScanner.scanTlds(TldScanner.java:350)
at org.apache.jasper.runtime.TldScanner.onStartup(TldScanner.java:239)
at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:106)
at org.eclipse.jetty.annotations.ServletContainerInitializerListener.doStart(ServletContainerInitializerListener.java:107)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.util.component.AggregateLifeCycle.doStart(AggregateLifeCycle.java:81)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:58)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:96)
at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:115)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1242)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:494)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:172)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95)
at org.eclipse.jetty.server.Server.doStart(Server.java:282)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at com.cloudera.nav.server.NavServer.run(NavServer.java:437)
at com.cloudera.nav.server.NavServer.main(NavServer.java:98)

 

2. The role logs show problems during the internal solr core startup.

2016-05-04 09:47:11,280 WARN org.apache.solr.update.UpdateLog: Starting log replay tlog{file=/var/lib/cloudera-scm-navigator/solr/<DIR>/data/<DIR>/log.0000000000000064928 refcount=2} active=false starting pos=0
2016-05-04 09:47:11,602 WARN org.apache.solr.update.UpdateLog: Starting log replay tlog{file=/var/lib/cloudera-scm-navigator/solr/<DIR>/data/<DIR>/log.0000000000000023308 refcount=2} active=false starting pos=0
2016-05-04 09:47:32,913 INFO com.cloudera.nav.server.NavServer: Navigator Metadata Server listening on http://<PRIVATE_IP_NAVIGATOR_METADATA_HOST>:7187
2016-05-04 09:47:32,916 ERROR org.apache.solr.update.UpdateLog: java.io.EOFException
2016-05-04 09:47:37,466 ERROR org.apache.solr.update.UpdateLog: java.io.EOFException

3. The associated stdout/stderr logs show low heap configuration resulting in OutOfMemoryException.

# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="/usr/lib64/cmf/service/common/killparent.sh"
# Executing /bin/sh -c "/usr/lib64/cmf/service/common/killparent.sh"...
Tue May 10 08:27:34 CEST 2016

 

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.