Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6482

Use block ID-based block layout on datanodes

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.6.0
    • datanode
    • None
    • Hide
      The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.
      Show
      The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

    Description

      Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location.

      An extension of the work in HDFS-3290.

      Attachments

        1. HDFS-6482.9.patch
          93 kB
          James Thomas
        2. hadoop-24-datanode-dir.tgz
          623 kB
          James Thomas
        3. HDFS-6482.8.patch
          882 kB
          James Thomas
        4. 6482-design.doc
          13 kB
          James Thomas
        5. HDFS-6482.7.patch
          477 kB
          James Thomas
        6. HDFS-6482.6.patch
          477 kB
          James Thomas
        7. HDFS-6482.5.patch
          476 kB
          James Thomas
        8. HDFS-6482.4.patch
          59 kB
          James Thomas
        9. HDFS-6482.3.patch
          33 kB
          James Thomas
        10. HDFS-6482.2.patch
          33 kB
          James Thomas
        11. HDFS-6482.1.patch
          33 kB
          James Thomas
        12. HDFS-6482.patch
          32 kB
          James Thomas

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HDFS-8791 block ID-based DN storage layout can be very slow for datanode on ext4

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-8782 Upgrade to block ID-based DN storage layout delays DN registration

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-10925 Compilation fails in native link0 function on Windows.

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-3290 Use a better local directory layout for the datanode

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-7648 Verify that HDFS blocks are in the correct datanode directories

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-6800 Support Datanode layout changes with rolling upgrade

        • Major - Major loss of function.
        • Closed
        is required by

        Bug - A problem which impairs or prevents the functions of the product. HDFS-6981 Fix DN upgrade with layout version change

        • Major - Major loss of function.
        • Closed

        New Feature - A new feature of the product, which has yet to be developed. HDFS-6581 Write to single replica in memory

        • Major - Major loss of function.
        • Closed
        relates to

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-6975 Consider removing heavyweight synchronization for FsDataset operations

        • Major - Major loss of function.
        • Open

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            james.thomas James Thomas
            james.thomas James Thomas
            Votes:
            0 Vote for this issue
            Watchers:
            21 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment