Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-385

Design a pluggable interface to place replicas of blocks in HDFS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0, 1.2.0, 1-win
    • None
    • None
    • Reviewed
    • New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas.

    Description

      The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models.

      Attachments

        1. BlockPlacementPluggable.txt
          16 kB
          Dhruba Borthakur
        2. BlockPlacementPluggable2.txt
          61 kB
          Dhruba Borthakur
        3. BlockPlacementPluggable3.txt
          70 kB
          Dhruba Borthakur
        4. BlockPlacementPluggable4.txt
          76 kB
          Dhruba Borthakur
        5. BlockPlacementPluggable4.txt
          79 kB
          Dhruba Borthakur
        6. BlockPlacementPluggable5.txt
          79 kB
          Dhruba Borthakur
        7. BlockPlacementPluggable6.txt
          77 kB
          Dhruba Borthakur
        8. BlockPlacementPluggable7.txt
          77 kB
          Dhruba Borthakur
        9. blockplacementpolicy-branch-1.patch
          28 kB
          Sumadhur Reddy Bolli
        10. blockplacementpolicy-branch-1-win.patch
          27 kB
          Sumadhur Reddy Bolli
        11. rat094.txt
          0.4 kB
          Michael Stack
        12. blockplacementpolicy2-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        13. blockplacementpolicy2-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli
        14. blockplacementpolicy3-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        15. blockplacementpolicy3-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-347 DFS read performance suboptimal when client co-located on nodes with data

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-629 Remove ReplicationTargetChooser.java along with fixing import warnings.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1094 Intelligent block placement policy to decrease probability of block loss

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1451 Extend the pluggable block placement policy interface to handle/allow file-level intelligence

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3564 Design enhancements to the pluggable blockplacementpolicy

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3649 Port HDFS-385 to branch-1-win

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              4 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: