[HADOOP-2158] hdfsListDirectory in libhdfs does not scale

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.2
• Assignee: Christian Kunz
• Reporter: Christian Kunz
• Created: 06/Nov/07 06:27
• Updated: 08/Jul/09 17:05
• Resolved: 13/Dec/07 23:09

Description

hdfsListDirectory makes one rpc call using deprecated fs.FileSystem.listPaths, and then two rpc calls for every entry in the returned array. When running a job with more than 3000 mappers each running a pipes application using libhdfs to scan a dfs directory with about 100-200 entries, this results in about 1M rpc calls to the namenode server overwhelming it.

hdfsListDirectory should call fs.FileSystem.listStatus instead.

I will submit a patch.