Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15288

Add Available Space Rack Fault Tolerant BPP

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.2, 3.4.0
    • 3.4.0, 3.3.2
    • block placement
    • None
    • Reviewed
    • Hide
      Added a new BlockPlacementPolicy: "AvailableSpaceRackFaultTolerantBlockPlacementPolicy" which uses the same optimization logic as the AvailableSpaceBlockPlacementPolicy along with spreading the replicas across maximum number of racks, similar to BlockPlacementPolicyRackFaultTolerant.
      The BPP can be configured by setting the blockplacement policy class as org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceRackFaultTolerantBlockPlacementPolicy
      Show
      Added a new BlockPlacementPolicy: "AvailableSpaceRackFaultTolerantBlockPlacementPolicy" which uses the same optimization logic as the AvailableSpaceBlockPlacementPolicy along with spreading the replicas across maximum number of racks, similar to BlockPlacementPolicyRackFaultTolerant. The BPP can be configured by setting the blockplacement policy class as org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceRackFaultTolerantBlockPlacementPolicy

    Description

      The Present AvailableSpaceBlockPlacementPolicy extends the Default Block Placement policy, which makes it apt for Replicated files. But not very efficient for EC files, which by default use. BlockPlacementPolicyRackFaultTolerant. So propose a to add new BPP having similar optimization as ASBPP where as keeping the spread of Blocks to max racks, i.e as RackFaultTolerantBPP.

      This could extend BlockPlacementPolicyRackFaultTolerant, rather than the BlockPlacementPOlicyDefault like ASBPP and keep other logics of optimization same as ASBPP

      Attachments

        1. HDFS-15288-Addendum-01.patch
          1 kB
          Ayush Saxena
        2. HDFS-15288-03.patch
          20 kB
          Ayush Saxena
        3. HDFS-15288-02.patch
          17 kB
          Ayush Saxena
        4. HDFS-15288-01.patch
          17 kB
          Ayush Saxena

        Issue Links

          Activity

            People

              ayushtkn Ayush Saxena
              ayushtkn Ayush Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: