Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-10530

BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy

    XMLWordPrintableJSON

    Details

      Description

      This issue was found by Takuya Fukudome.

      Under RS-DEFAULT-6-3-64k EC policy,
      1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) of the cluster.
      2. Reconstruction work would be scheduled if the 6th rack is added.
      3. While adding the 7th rack or more racks will not trigger reconstruction work.

      Based on default EC block placement policy defined in “BlockPlacementPolicyRackFaultTolerant.java”, EC file should be able to be scheduled to distribute to 9 racks if possible.

      In BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock) , numReplicas of striped blocks might should be getRealTotalBlockNum(), instead of getRealDataBlockNum().

        Attachments

        1. HDFS-10530.1.patch
          5 kB
          Rui Gao
        2. HDFS-10530.2.patch
          8 kB
          Rui Gao
        3. HDFS-10530.3.patch
          7 kB
          Rui Gao
        4. HDFS-10530.4.patch
          9 kB
          Manoj Govindassamy
        5. HDFS-10530.5.patch
          10 kB
          Manoj Govindassamy

          Activity

            People

            • Assignee:
              manojg Manoj Govindassamy
              Reporter:
              demongaorui Rui Gao
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: