Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-385

Design a pluggable interface to place replicas of blocks in HDFS

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0, 1.2.0, 1-win
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas.

      Description

      The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models.

        Attachments

        1. rat094.txt
          0.4 kB
          stack
        2. blockplacementpolicy-branch-1-win.patch
          27 kB
          Sumadhur Reddy Bolli
        3. blockplacementpolicy-branch-1.patch
          28 kB
          Sumadhur Reddy Bolli
        4. blockplacementpolicy3-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli
        5. blockplacementpolicy3-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        6. blockplacementpolicy2-branch-1-win.patch
          84 kB
          Sumadhur Reddy Bolli
        7. blockplacementpolicy2-branch-1.patch
          85 kB
          Sumadhur Reddy Bolli
        8. BlockPlacementPluggable7.txt
          77 kB
          dhruba borthakur
        9. BlockPlacementPluggable6.txt
          77 kB
          dhruba borthakur
        10. BlockPlacementPluggable5.txt
          79 kB
          dhruba borthakur
        11. BlockPlacementPluggable4.txt
          76 kB
          dhruba borthakur
        12. BlockPlacementPluggable4.txt
          79 kB
          dhruba borthakur
        13. BlockPlacementPluggable3.txt
          70 kB
          dhruba borthakur
        14. BlockPlacementPluggable2.txt
          61 kB
          dhruba borthakur
        15. BlockPlacementPluggable.txt
          16 kB
          dhruba borthakur

          Issue Links

            Activity

              People

              • Assignee:
                dhruba dhruba borthakur
                Reporter:
                dhruba dhruba borthakur
              • Votes:
                4 Vote for this issue
                Watchers:
                37 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: