TaskTracker In Bad State on Oracle Big Data Appliance with Error "The DiskSpace quota of /user/(username) is Exceeded" (Doc ID 1573575.1)

Last updated on JANUARY 23, 2014

Applies to:

Big Data Appliance Integrated Software - Version 2.0.1 and later
Linux x86-64

Symptoms

A TaskTracker on an Oracle Big Data Appliance (BDA) node goes down and will not restart. The TaskTracker shows "Bad" health in Cloudera Manager(CM). The TASKTRACKER log file at:

/var/log/hadoop-0.20-mapreduce/hadoop-cmf-mapreduce-TASKTRACKER-<node name.example.com>.log.out shows a DSQuotaExceededException like:

2013-07-07 12:25:45,797 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201307051138_0107_r_820472470 exited with exit code 0. Number of tasks it ran: 0
2013-07-07 12:25:45,817 INFO org.apache.hadoop.mapred.TaskStatus: task-diagnostic-info for task attempt_201307051138_0107_r_000049_0 : org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/<username> is exceeded: quota=20.0g diskspace consumed=20.7g
at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:161)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1632)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1373)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:347)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:2436)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2191)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms