Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13788

Update EC documentation about rack fault tolerance

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      From http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html:

      For rack fault-tolerance, it is also important to have at least as many racks as the configured EC stripe width. For EC policy RS (6,3), this means minimally 9 racks, and ideally 10 or 11 to handle planned and unplanned outages. For clusters with fewer racks than the stripe width, HDFS cannot maintain rack fault-tolerance, but will still attempt to spread a striped file across multiple nodes to preserve node-level fault-tolerance.

      Theoretical minimum is 3 racks, and ideally 9 or more, so the document should be updated.

      (I didn't check timestamps, but this is probably due to BlockPlacementPolicyRackFaultTolerant isn't completely done when HDFS-9088 introduced this doc. Later there's also examples in TestErasureCodingMultipleRacks to test this explicitly.)

      Attachments

        1. HDFS-13788.002.patch
          2 kB
          Kitti Nanasi
        2. HDFS-13788.001.patch
          2 kB
          Kitti Nanasi

        Activity

          People

            knanasi Kitti Nanasi
            xiaochen Xiao Chen
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: