[HADOOP-6963] Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.20.205.0, 0.23.1
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: fs
• Target Version/s: 1.0.3, 0.23.3
• Hadoop Flags: Reviewed
• Assignee: Ravi Prakash
• Reporter: Owen O'Malley
• Created: 20/Sep/10 07:27
• Updated: 07/Sep/12 21:01
• Resolved: 05/Apr/12 19:51

Description

The getDU method should not include the size of the directory. The Java interface says that the value is undefined and in Linux/Sun it gets the 4096 for the inode. Clearly this isn't useful.
It also recursively calls itself. In case the directory has a symbolic link forming a cycle, getDU keeps spinning in the cycle. In our case, we saw this in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects call. This prevented other tasks on the same node from committing, causing the TT to become effectively useless (because the JT thinks it already has enough tasks running)