[HDFS-13616] Batch listing of multiple directories

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Assignee: Chao Sun
• Reporter: Andrew Wang
• Created: 24/May/18 21:10
• Updated: 02/Mar/20 17:12
• Resolved: 16/Jan/20 04:23

Description

One of the dominant workloads for external metadata services is listing of partition directories. This can end up being bottlenecked on RTT time when partition directories contain a small number of files. This is fairly common, since fine-grained partitioning is used for partition pruning by the query engines.

A batched listing API that takes multiple paths amortizes the RTT cost. Initial benchmarks show a 10-20x improvement in metadata loading performance.

Issue Links

• is related to: HADOOP-16898 Batch listing of multiple directories to be an unstable interface
• relates to: HDFS-14233 Implement DistributedFileSystem#listStatus(Path[]) by adding a batching listStatus RPC call to NameNode
• links to: GitHub Pull Request #1725

Sub-Tasks

• RBF: Supporting batched listing: Open
• Batch listing: gracefully fallback to use non-batched listing when NameNode doesn't support the feature: Patch Available