My Oracle Support Banner

Upgrade Mammoth to V4.1.0 Fails in Step 7 When Upgrading the Sqoop 2 Service with: Failed to Upgrade Sqoop (Doc ID 1966882.1)

Last updated on AUGUST 03, 2021

Applies to:

Big Data Appliance Integrated Software - Version 3.0.1 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

Upgrading from BDA Mammoth v3.0.1/CDH 5.0.1 to BDA Mammoth v4.1.0/CDH 5.3 fails to upgrade the Sqoop 2 service.  The Mammoth upgrade from 3.0.1 to 4.1.0 raises the following error occur on step 7 of Mammoth script:

ERROR: Puppet agent run on node bdanode04 had errors. List of errors follows

************************************
Error [10641]: (//bdanode04.example.com//Stage[main]/Hadoop::Startcloudera/Exec[start_cloudera_services]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/startcloudera_1422662582.out returned 1 instead of one of [0]
************************************

 

The Sqoop 2 service shows bad health in Cloudera Manager (CM) and clicking on Sqoop 2 service shows the Sqoop 2 Server is "down."

Added Sqoop 2 to Hue as per the last step in "How to Configure the Sqoop 2 Service in Cloudera Manager (CM) for Oracle Big Data Appliance 3.1.0/4.X (Doc ID 1922701.1)" however the Sqoop 2 service does not start.

Re-running mammoth -p fails with:

ERROR: Puppet agent run on node bdanode04 had errors. List of errors follows

************************************
Error [10641]: (//bdanode04.example.com//Stage[main]/Hadoop::Upgradecdh/Exec[upgrade_cdh]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/upgradecdhparcel.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/upgradecdhparcel.out returned 1 instead of one of [0]
************************************

 

From the Sqoop 2 service in CM you can manually run the "Create Sqoop User Directory" menu option but selecting the options of "Upgrade Sqoop" or "Create Sqoop Database"  fail.

When trying to perform the following steps the logs below show the errors:


1. CM > Commands > Upgrade Sqoop

Failed to Upgrade Sqoop


Details stderr.log

+ CATALINA_OPTS='-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/log4j.properties -Dsqoop.config.dir=/var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005 '
+ export LOGGING_CONFIG=-Djava.util.logging.config.file=/var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/catalina_logging.properties
+ LOGGING_CONFIG=-Djava.util.logging.config.file=/var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/catalina_logging.properties
+ mkdir -p /var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/temp
+ EXT_LIBS_DIR=/var/lib/sqoop2
+ cp /usr/share/java/mysql-connector-java.jar /var/lib/sqoop2
+ cp /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar /var/lib/sqoop2
+ cp /usr/share/java/oracle-connector-java.jar /var/lib/sqoop2
+ '[' upgrade == upgrade -o upgrade == create-database ']'
+ export 'JAVA_OPTS=-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/log4j.properties -Dsqoop.config.dir=/var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005 '
+ JAVA_OPTS='-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade/log4j.properties -Dsqoop.config.dir=/var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005 '
+ exec /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/bigtop-tomcat/bin/tool-wrapper.sh -server org.apache.sqoop.tomcat.TomcatToolRunner upgrade

 

Upgrade Sqoop 2 Server
Process sqoop-server-upgrade (id=738) on host bdanode04.example.com (id=3) exited with 1 and expected 0




From /var/log/sqoop2/sqoop-cmf-sqoop-SQOOP_SERVER-bdanode04.example.com.log.out:

2015-02-02 12:04:47,921 ERROR org.apache.sqoop.tools.tool.UpgradeTool: Can't finish upgrading RepositoryManager, Driver and ConnectionManager:
org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)
at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Tool.main(Tool.java:225)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:derby:/var/lib/sqoop2/repository/db;create=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:51)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)
... 17 more
2015-02-02 12:04:47,923 INFO org.apache.sqoop.core.PropertiesConfigurationProvider: Shutting down configuration poller thread


From /var/run/cloudera-scm-agent/process/738-sqoop-server-upgrade stdout.log:

# more stdout.log
Mon Feb 2 12:04:47 EST 2015
JAVA_HOME=/usr/java/jdk1.7.0_72
using 5 as CDH_VERSION
Sqoop tool executor:
Version: 1.99.4-cdh5.3.0
Revision: 75a4ffb64ddc4001a26a04366271b51b262224a1
Compiled on Tue Dec 16 20:14:23 PST 2014 by jenkins
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
Upgrade has failed, please check Server logs for further details.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has failed.



2. Create Sqoop Database

Failed to create Sqoop database.

Program: sqoop/sqoop.sh ["create-database"]

From /var/run/cloudera-scm-agent/process/739-sqoop-create-database

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.