On Oracle Big Data Appliance Kerberos Enabled Cluster both Hive/Sentry Service are Down After CM Generated Principal Password is Manually Changed (Doc ID 2007378.1)

Last updated on OCTOBER 11, 2016

Applies to:

Big Data Appliance Integrated Software - Version 2.5.0 and later
Linux x86-64

Symptoms

On Oracle Big Data Appliance Hive Server reported failure of Canary test .

Hive Metastore Server logs indicate below failure

10:13:46.501 PM ERROR org.apache.sentry.hdfs.MetastorePlugin
Error talking to Sentry HDFS Service !!
java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
............
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: sentry.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
................
at org.apache.sentry.hdfs.SentryHDFSServiceClient$UgiSaslClientTransport$1.run(SentryHDFSServiceClient.java:105)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
... 12 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sentry.org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 21 more

10:13:47.502 PM INFO org.apache.sentry.hdfs.SentryHDFSServiceClient

Using server kerberos principal: sentry/bdanode04****@<RealmName>

10:13:47.503 PM WARN org.apache.hadoop.security.UserGroupInformation

PriviledgedActionException as:hive/bdanode04******@<RealmName> (auth:KERBEROS) cause:sentry.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

10:13:47.503 PM ERROR org.apache.sentry.hdfs.MetastorePlugin

Error talking to Sentry HDFS Service !!
java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
at org.apache.sentry.hdfs.SentryHDFSServiceClient$UgiSaslClientTransport.open(SentryHDFSServiceClient.java:105)
at org.apache.sentry.hdfs.SentryHDFSServiceClient.(SentryHDFSServiceClient.java:167)

Sentry Server is Stopped and trying to start Sentry service fails with below errors

Sentry Server
10:07:41.861 PM ERROR org.apache.sentry.service.thrift.SentryService

Error starting server
javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:763)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
....................
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:177)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:149)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:288)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735)
... 20 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
... 27 more

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms