On BDA 4.2/CDH 5.4.4 Too Many Yarn Logs are Stored in HDFS in /tmp/logs/{user}/logs and JobHistory Logs Report "java.lang.IllegalArgumentException: Failed to specify server's Kerberos" (Doc ID 2130142.1)

Last updated on APRIL 25, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

On BDA 4.2/CDH 5.4.4 too many Yarn logs, specifically Hive application logs are found to be stored in HDFS in /tmp/logs/{user}/logs.  The file retention appears to be infinite.  Since cluster creation it is observed the 850 GB of logs exist.

1. It is observed in Cloudera Manager that the Yarn configuration parameter "Log Aggregation Retention Period" is set to 7 days.  But that does not appear to be working.

2. The yarn-site.xml file created by Cloudera Manager and distributed to the cluster shows:
 

yarn.log-aggregation.retain-seconds

604800 

 
This value refers to 7 days. So again the 7 day retention period does not seem to be picked up by the system.


3. The Job History Server logs report exceptions like below:


2016-04-18 23:16:43,831 INFO org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking getApplicationReport of class
ApplicationClientProtocolPBClientImpl over rm15 after 7332 fail over attempts. Trying to fail over immediately. java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos
principal name; Host Details : local host is: "node3.example.com/192.xxx.x.x"; destination host is:
"node3.example.com":8032;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399)

Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680) at java.security.AccessController.doPrivileged(Native Method)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms