Step 10 Fails Running Mammoth -e Expanding a v4.2.0 Oracle Big Data Appliance Cluster When Network Encryption Enabled For the Cluster or if HTTPS is Enabled for Cloudera Manager (Doc ID 2143756.1)

Last updated on AUGUST 10, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

Step 10 Fails with the following during expansion of Oracle Big Data Appliance v4.2.0:

INFO: Step 10 StartHadoopServices
INFO: Acquiring run lock ...
INFO: Creating nodelist files...
INFO: Creating environment.pp file ...
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents
INFO: Copying JCE policy files. This will take some time ...
INFO: Puppet script successfully sent to node bdanode07
INFO: Puppet script successfully sent to node bdanode08
INFO: Puppet script successfully sent to node bdanode09
INFO: Puppet script successfully sent to node bdanode10
INFO: Puppet script successfully sent to node bdanode11
INFO: Puppet script successfully sent to node bdanode12
INFO: Puppet script successfully sent to node bdanode13
INFO: Puppet script successfully sent to node bdanode14
INFO: Puppet script successfully sent to node bdanode15
INFO: Pinging puppet agents ..
INFO: Puppet agent run on node bdanode07 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode07-20160525181258.log
INFO: Puppet agent run on node bdanode08 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode08-20160525181258.log
INFO: Puppet agent run on node bdanode09 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode09-20160525181258.log
INFO: Puppet agent run on node bdanode10 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode10-20160525181258.log
INFO: Puppet agent run on node bdanode11 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode11-20160525181258.log
INFO: Puppet agent run on node bdanode12 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode12-20160525181258.log
INFO: Puppet agent run on node bdanode13 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode13-20160525181258.log
INFO: Puppet agent run on node bdanode14 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode14-20160525181258.log
INFO: Puppet agent run on node bdanode15 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode15-20160525181258.log
SUCCESS: Successfully copied JCE policy files.
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents
INFO: Starting Hadoop Services. This will take some time ...
INFO: Puppet script successfully sent to node bdanode01
INFO: Puppet script successfully sent to node bdanode07
INFO: Puppet script successfully sent to node bdanode08
INFO: Puppet script successfully sent to node bdanode09
INFO: Puppet script successfully sent to node bdanode10
INFO: Puppet script successfully sent to node bdanode11
INFO: Puppet script successfully sent to node bdanode12
INFO: Puppet script successfully sent to node bdanode13
INFO: Puppet script successfully sent to node bdanode14
INFO: Puppet script successfully sent to node bdanode15
INFO: Pinging puppet agents ...........................................................................................................
ERROR: Puppet agent run on node bdanode01 had errors. List of errors follows

************************************
Error [1093]: (//bdanode01.example.com//Stage[main]/Hadoop::Startsvc2/Exec[setup_scm]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/setupscm.sh &> /opt/oracle/BDAMammoth/bdaconfig/tmp/setupscm_1464214330.out returned 1 instead of one of [0]
************************************

INFO: Also check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode01-20160525181504.log

INFO: Puppet agent run on node bdanode07 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode07-20160525181504.log
INFO: Puppet agent run on node bdanode08 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode08-20160525181504.log
INFO: Puppet agent run on node bdanode09 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode09-20160525181504.log
INFO: Puppet agent run on node bdanode10 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode10-20160525181504.log
INFO: Puppet agent run on node bdanode11 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode11-20160525181504.log
INFO: Puppet agent run on node bdanode12 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode12-20160525181504.log
INFO: Puppet agent run on node bdanode13 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode13-20160525181504.log
INFO: Puppet agent run on node bdanode14 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode14-20160525181504.log
INFO: Puppet agent run on node bdanode15 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode15-20160525181504.log
ERROR: Puppet script had errors.
ERROR: unable to continue with install until errors are fixed #Step 10#
INFO: Time spent in step 10 StartHadoopServices is 1396 seconds.
INFO: Running bdadiagcluster...
INFO: Cloudera Manager is not accessible - cannot get service state. Skipping running dumpcluster
INFO: Starting Big Data Appliance diagnose cluster at Wed May 25 18:35:24 2016
INFO: Logging results to /tmp/bda_diagcluster_1464215720.log

 

The /opt/oracle/BDAMammoth/bdaconfig/tmp/setupscm_1464214330.out shows:

Operation failed
Result Message is: "Command 'Start' failed for cluster 'bdaclu'",

 

In /var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-bdanode15.example.com.log.out see:

Exception in secureMain
java.io.FileNotFoundException: /opt/cloudera/security/jks/keystore.jks (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
at org.mortbay.jetty.security.SslSocketConnector.createFactory(SslSocketConnector.java:242)
at org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:476)
at org.apache.hadoop.security.ssl.SslSocketConnectorSecure.newServerSocket(SslSocketConnectorSecure.java:46)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:887)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:828)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:795)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1145)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:430)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2404)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2291)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2338)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2515)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms