[HADOOP-713] dfs list operation is too expensive

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.15.1
• Assignee: Dhruba Borthakur
• Reporter: Hairong Kuang
• Created: 13/Nov/06 23:24
• Updated: 08/Jul/09 16:42
• Resolved: 19/Nov/07 20:08

Description

A list request to dfs returns an array of DFSFileInfo. A DFSFileInfo of a directory contains a field called contentsLen, indicating its size which gets computed at the namenode side by resursively going through its subdirs. At the same time, the whole dfs directory tree is locked.

The list operation is used a lot by DFSClient for listing a directory, getting a file's size and # of replicas, and getting the size of dfs. Only the last operation needs the field contentsLen to be computed.

To reduce its cost, we can add a flag to the list request. ContentsLen is computed If the flag is set. By default, the flag is false.

Issue Links

• relates to: HADOOP-909 dfs "du" shows that the size of a subdirectory is 0