Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-385

Design a pluggable interface to place replicas of blocks in HDFS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0, 1.2.0, 1-win
    • None
    • None
    • Reviewed
    • New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas.

    Description

      The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models.

      Attachments

        1. BlockPlacementPluggable.txt
          16 kB
          Dhruba Borthakur
        2. BlockPlacementPluggable2.txt
          61 kB
          Dhruba Borthakur
        3. BlockPlacementPluggable3.txt
          70 kB
          Dhruba Borthakur
        4. BlockPlacementPluggable4.txt
          79 kB
          Dhruba Borthakur
        5. BlockPlacementPluggable4.txt
          76 kB
          Dhruba Borthakur
        6. BlockPlacementPluggable5.txt
          79 kB
          Dhruba Borthakur
        7. BlockPlacementPluggable6.txt
          77 kB
          Dhruba Borthakur
        8. BlockPlacementPluggable7.txt
          77 kB
          Dhruba Borthakur
        9. blockplacementpolicy2-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        10. blockplacementpolicy2-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli
        11. blockplacementpolicy3-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        12. blockplacementpolicy3-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli
        13. blockplacementpolicy-branch-1.patch
          28 kB
          Sumadhur Reddy Bolli
        14. blockplacementpolicy-branch-1-win.patch
          27 kB
          Sumadhur Reddy Bolli
        15. rat094.txt
          0.4 kB
          Michael Stack

        Issue Links

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              4 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: