[HDFS-3290] Use a better local directory layout for the datanode

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.23.0
• Target Version/s: 2.0.0-alpha
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 17/Apr/12 20:22
• Updated: 15/Jan/16 23:08
• Resolved: 15/Jan/16 23:08

Description

When the HDFS DataNode stores chunks in a local directory, it currently puts all of the chunk files into either one big directory, or a collection of directories. However, there is no way to know which directory a given block will end up in, given its ID. As the number of files increases, this does not scale well.

Similar to the git version control system, HDFS should create a few different top level directories keyed off of a few bits in the chunk ID. Git uses 8 bits. This substantially cuts down on the number of chunk files in the same directory and gives increased performance, while not compromising O(1) lookup of chunks.

Issue Links

• duplicates: HDFS-6482 Use block ID-based block layout on datanodes