Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6482

Use block ID-based block layout on datanodes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Release Note:
      Hide
      The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.
      Show
      The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

      Description

      Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location.

      An extension of the work in HDFS-3290.

        Attachments

        1. HDFS-6482.patch
          32 kB
          James Thomas
        2. HDFS-6482.1.patch
          33 kB
          James Thomas
        3. HDFS-6482.2.patch
          33 kB
          James Thomas
        4. HDFS-6482.3.patch
          33 kB
          James Thomas
        5. HDFS-6482.4.patch
          59 kB
          James Thomas
        6. HDFS-6482.5.patch
          476 kB
          James Thomas
        7. HDFS-6482.6.patch
          477 kB
          James Thomas
        8. HDFS-6482.7.patch
          477 kB
          James Thomas
        9. 6482-design.doc
          13 kB
          James Thomas
        10. HDFS-6482.8.patch
          882 kB
          James Thomas
        11. hadoop-24-datanode-dir.tgz
          623 kB
          James Thomas
        12. HDFS-6482.9.patch
          93 kB
          James Thomas

          Issue Links

            Activity

              People

              • Assignee:
                james.thomas James Thomas
                Reporter:
                james.thomas James Thomas
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: