My Oracle Support Banner

Huge Number of Files Accumulating in /var/log/hadoop-hdfs/audit on Directory on BDA V4.2/CDH 5.4. (Doc ID 2046942.1)

Last updated on DECEMBER 19, 2019

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64


NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

The Navigator Audit Server Audit log directory, the directory in which audit event log files are written for HDFS, /var/log/hadoop-hdfs/audit, contains a huge number of files.

This is observed on one or more nodes of the cluster.  Listing the files in /var/log/hadoop-hdfs/audit looks like:

-rw-r--r-- 1 hdfs hadoop 104857664 Jun 12 09:42 hdfs-NAMENODE-6d61dd753dff6dd467ac6866d5fd7203-HdfsAuditLogger-1440355964600
-rw-r--r-- 1 hdfs hadoop 104857755 Jun 12 13:56 hdfs-NAMENODE-6d61dd753dff6dd467ac6866d5fd7203-HdfsAuditLogger-14403559646**
-rw-r--r-- 1 hdfs hadoop 104857823 Jun 12 18:05 hdfs-NAMENODE-6d61dd753dff6dd467ac6866d5fd7203-HdfsAuditLogger-14403559646**
-rw-r--r-- 1 hdfs hadoop 104857624 Jun 12 22:15 hdfs-NAMENODE-6d61dd753dff6dd467ac6866d5fd7203-HdfsAuditLogger-14403559646**
... // hundreds of files

From Cloudera Manager it is confirmed that the maximum audit log file size is the default of 100MB and the maximum audit logs to retain is the default of 10.

The HDFS audit log directory properties can be verified in CM by navigating:

For Audit Log Directory/audit_event_log_dir:
home > hdfs > Configuration > Search: audit_event_log_dir > Value is /var/log/hadoop-hdfs/audit.

For Maximum Audit Log File Size/navigator.audit_log_max_file_size:
home > hdfs > Configuration > Search: navigator.audit_log_max_file_size > Value is 100MB.

For Maximum Audit Logs to Retain/navigator.audit_log_max_backup_index
home > hdfs > Configuration > Search: navigator.audit_log_max_backup_index > Value is 10.



To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.