[HDFS-3061] Backport HDFS-1487 to branch-1

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.20.203.0, 0.23.0, 1.0.2
• Fix Version/s: 1.0.3
• Component/s: namenode

Environment

0.20.203 with HDFS-1377 and HDFS-2053 patches applied

• Assignee: Kihwal Lee
• Reporter: Alex Holmes
• Created: 08/Mar/12 00:11
• Updated: 12/Jun/12 05:30
• Resolved: 06/May/12 20:39

Description

It appears that there's a condition under which a HDFS directory with a space quota set can get to a point where the cached size for the directory can permanently differ from the computed value. When this happens the following command:

hadoop fs -count -q /tmp/quota-test

results in the following output in the NameNode logs:

WARN org.apache.hadoop.hdfs.server.namenode.NameNode: Inconsistent diskspace for directory quota-test. Cached: 6000 Computed: 6072

I've observed both transient and persistent instances of this happening. In the transient instances this warning goes away, but in the persistent instances every invocation of the fs -count -q command yields the above warning.

I've seen instances where the actual disk usage of a directory is 25% of the cached value in INodeDirectory, which creates problems since the quota code uses this cached value to determine whether block write requests are permitted.

This isn't easy to reproduce - I am able to (inconsistently) get HDFS into this state with a simple program which:

1. Writes files into HDFS
2. When a DSQuotaExceededException is encountered removes all files created in step 1
3. Repeat step 1

I'm going to try and come up with a more repeatable test case to reproduce this issue.

Issue Links

• is related to: HDFS-1487 FSDirectory.removeBlock() should update diskspace count of the block owner node
• relates to: HDFS-1377 Quota bug for partial blocks allows quotas to be violated
• relates to: HDFS-2053 Bug in INodeDirectory#computeContentSummary warning