After Installing Keytrustee 5.3.0 Jobs Raise "java.net.ConnectException: Connection refused" (Doc ID 2091808.1)

Last updated on DECEMBER 24, 2015

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

All jobs run as any user fail after installing/configuring Keytrustee 5.3.0 on a BDA V4.2/CDH 5.4.4 cluster as below:

[oracle@bdanode01 ~]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.4.4.jar pi 10 1000
Number of Maps = 10
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
15/12/21 14:17:49 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 2910 for oracle on ha-hdfs:bdaclu-ns
15/12/21 14:17:50 WARN security.UserGroupInformation: PriviledgedActionException as:oracle@BDA.EXAMPLE.COM (auth:KERBEROS) cause:java.net.ConnectException: Connection refused
15/12/21 14:17:50 WARN security.UserGroupInformation: PriviledgedActionException as:oracle@BDA.EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: java.net.ConnectException: Connection refused
java.io.IOException: java.net.ConnectException: Connection refused
  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
  at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
  at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2056)
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
  at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:142)
  at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
  at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
  at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1324)
  at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
  at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:497)
  at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
  at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
  at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:497)
  at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Connection refused
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketI

 

The kms-catalina.2015-12-18.log complains about permission denied on keystore.jks.  For example:

java.io.FileNotFoundException: /opt/cloudera/security/jks/keystore.jks (Permission denied) 

 

The commands 'hadoop key list' and 'hadoop create key' do not work. The command 'hadoop create key' throws an error in the kms.log saying the password is tampered with or incorrect for keystore:

2015-12-21 18:48:49,866 DEBUG com.cloudera.keytrustee.TrusteeKeyProvider: getKeyVersions threw KeyTrusteeException
com.cloudera.keytrustee.KeyTrusteeException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at com.cloudera.keytrustee.impl.ClientConnectionImpl.getDepositsJSON(ClientConnectionImpl.java:949)
at com.cloudera.keytrustee.impl.ClientConnectionImpl.getDeposits(ClientConnectionImpl.java:899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloudera.keytrustee.ClientConnectionInvocationHandler.invoke(ClientConnectionInvocationHandler.java:46)
at com.sun.proxy.$Proxy5.getDeposits(Unknown Source)
at com.cloudera.keytrustee.TrusteeKeyProvider.getAllDeposits(TrusteeKeyProvider.java:982)
at com.cloudera.keytrustee.TrusteeKeyProvider.getHandleDeposits(TrusteeKeyProvider.java:927)
at com.cloudera.keytrustee.TrusteeKeyProvider.getKeyVersions(TrusteeKeyProvider.java:400)
at com.cloudera.keytrustee.TrusteeKeyProvider.keyExists(TrusteeKeyProvider.java:682)
at com.cloudera.keytrustee.TrusteeKeyProvider.createKey(TrusteeKeyProvider.java:307)
at org.apache.hadoop.crypto.key.KeyProvider.createKey(KeyProvider.java:515)
at org.apache.hadoop.crypto.key.KeyProviderExtension.createKey(KeyProviderExtension.java:74)
at org.apache.hadoop.crypto.key.KeyProviderExtension.createKey(KeyProviderExtension.java:74)
at org.apache.hadoop.crypto.key.KeyProviderExtension.createKey(KeyProviderExtension.java:74)
at org.apache.hadoop.crypto.key.kms.server.KeyAuthorizationKeyProvider.createKey(KeyAuthorizationKeyProvider.java:160)
at org.apache.hadoop.crypto.key.kms.server.KMS$1.run(KMS.java:135)
at org.apache.hadoop.crypto.key.kms.server.KMS$1.run(KMS.java:132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.crypto.key.kms.server.KMS.createKey(KMS.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555)
at org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:255)
at sun.net.www.protocol.https.HttpsClient.createSocket(HttpsClient.java:405)
at sun.net.NetworkClient.doConnect(NetworkClient.java:162)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:163)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:120)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:114)
at com.cloudera.keytrustee.impl.EncryptedPayloadConnection.fetchData(EncryptedPayloadConnection.java:78)
at com.cloudera.keytrustee.impl.ClientConnectionImpl.getDepositsJSON(ClientConnectionImpl.java:945)
... 64 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1617)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122)
at javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:332)
at javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:289)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:85)
at sun.net.www.protocol.https.Handler.openConnection(Handler.java:62)
at sun.net.www.protocol.https.Handler.openConnection(Handler.java:57)
at java.net.URL.openConnection(URL.java:972)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:131)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:120)
at com.cloudera.keytrustee.URLConnector.execQuery(URLConnector.java:114)
at com.cloudera.keytrustee.impl.EncryptedPayloadConnection.fetchData(EncryptedPayloadConnection.java:78)
at com.cloudera.keytrustee.impl.ClientConnectionImpl.getDepositsJSON(ClientConnectionImpl.java:945)
at com.cloudera.keytrustee.impl.ClientConnectionImpl.getDeposits(ClientConnectionImpl.java:899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloudera.keytrustee.ClientConnectionInvocationHandler.invoke(ClientConnectionInvocationHandler.java:46)
at com.sun.proxy.$Proxy5.getDeposits(Unknown Source)
at com.cloudera.keytrustee.TrusteeKeyProvider.getAllDeposits(TrusteeKeyProvider.java:982)
at com.cloudera.keytrustee.TrusteeKeyProvider.getAllDeposits(TrusteeKeyProvider.java:905)
at com.cloudera.keytrustee.TrusteeKeyProvider.getAllDeposits(TrusteeKeyProvider.java:897)
at com.cloudera.keytrustee.TrusteeKeyProvider.getKeys(TrusteeKeyProvider.java:519)
at org.apache.hadoop.crypto.key.KeyProviderExtension.getKeys(KeyProviderExtension.java:90)
at org.apache.hadoop.crypto.key.KeyProviderExtension.getKeys(KeyProviderExtension.java:90)
at org.apache.hadoop.crypto.key.KeyProviderExtension.getKeys(KeyProviderExtension.java:90)
at org.apache.hadoop.crypto.key.kms.server.KeyAuthorizationKeyProvider.getKeys(KeyAuthorizationKeyProvider.java:283)
at org.apache.hadoop.crypto.key.kms.server.KMS$5.run(KMS.java:259)
at org.apache.hadoop.crypto.key.kms.server.KMS$5.run(KMS.java:256)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.crypto.key.kms.server.KMS.getKeyNames(KMS.java:255)
... 41 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultTrustManager(SSLContextImpl.java:767)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:733)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)
... 78 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 90 more

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms