[HDFS-6482] Use block ID-based block layout on datanodes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: datanode

Release Note

The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

• Assignee: James Thomas
• Reporter: James Thomas
• Created: 03/Jun/14 20:49
• Updated: 12/May/16 18:11
• Resolved: 01/Aug/14 20:41

Description

Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location.

An extension of the work in HDFS-3290.

Issue Links

• breaks: HDFS-8791 block ID-based DN storage layout can be very slow for datanode on ext4
• breaks: HDFS-8782 Upgrade to block ID-based DN storage layout delays DN registration
• breaks: HADOOP-10925 Compilation fails in native link0 function on Windows.
• is duplicated by: HDFS-3290 Use a better local directory layout for the datanode
• is related to: HDFS-7648 Verify that HDFS blocks are in the correct datanode directories
• is related to: HDFS-6800 Support Datanode layout changes with rolling upgrade
• is required by: HDFS-6981 Fix DN upgrade with layout version change
• is required by: HDFS-6581 Write to single replica in memory
• relates to: HDFS-6975 Consider removing heavyweight synchronization for FsDataset operations