Use block ID-based block layout on datanodes

XMLWordPrintableJSON

Details

• Improvement
• Status: Closed
• Major
• Resolution: Fixed
• 3.0.0-alpha1
• None
• Hide
The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.
Show
The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

Description

Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location.

An extension of the work in HDFS-3290.

Attachments

1. 6482-design.doc
13 kB
James Thomas
623 kB
James Thomas
3. HDFS-6482.1.patch
33 kB
James Thomas
4. HDFS-6482.2.patch
33 kB
James Thomas
5. HDFS-6482.3.patch
33 kB
James Thomas
6. HDFS-6482.4.patch
59 kB
James Thomas
7. HDFS-6482.5.patch
476 kB
James Thomas
8. HDFS-6482.6.patch
477 kB
James Thomas
9. HDFS-6482.7.patch
477 kB
James Thomas
10. HDFS-6482.8.patch
882 kB
James Thomas
11. HDFS-6482.9.patch
93 kB
James Thomas
12. HDFS-6482.patch
32 kB
James Thomas

People

James Thomas
James Thomas