After BDA v4.1 Upgrade with Kerberos and Sentry Authorization via Policy Files, Oozie Java Action Fails with"token (HDFS_DELEGATION_TOKEN token 5236 for u_m0red) can't be found in cache" (Doc ID 2014294.1)

Last updated on MAY 27, 2015

Applies to:

Big Data Appliance Integrated Software - Version 4.1.0 and later
Linux x86-64

Symptoms

After BDA V4.0 (CDH 5.1.2) to BDA V4.1 (CDH 5.3.0) upgrade in an environment with Kerberos and Sentry Authorization via Policy Files, a Java action from oozie fails as below. The jobs are run directly on the BDA. Oozie jobs are submitted in Hue. The same job worked successfully prior to upgrade. 

1. JA017 appears in Hue on an Oozie Action:

JA017: Unknown hadoop job [job_xxx_xxx] associated with action [xxx-xxx-oozie-oozi-W@stage].



2. Log shows: token (HDFS_DELEGATION_TOKEN token 5236 for u_m0red) can't be found in cache.


2015-04-29 17:05:44,831 WARN [LeaseRenewer:u_m0red@bda21] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 5236 for u_m0red) can't be found in cache
2015-04-29 17:05:44,831 WARN [LeaseRenewer:u_m0red@bda21] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:u_m0red (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 5236 for u_m0red) can't be found in cache
2015-04-29 17:05:44,838 WARN [LeaseRenewer:u_m0red@bda21] org.apache.hadoop.hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_attempt_1429864641581_1347_m_000000_0_-998114578_1] for 30 seconds.  Will retry shortly ...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 5236 for u_m0red) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:563)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:849)
at org.apache.hadoop.hdfs.Leas



Changes

 Upgrade from BDA V4.0 (CDH 5.1.2) to BDA V4.1 (CDH 5.3.0)

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms