Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1172

Blocks in newly completed files are considered under-replicated too quickly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

        Attachments

        1. HDFS-1172.patch
          3 kB
          Boris Shkolnik
        2. replicateBlocksFUC.patch
          4 kB
          Hairong Kuang
        3. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        4. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        5. hdfs-1172.txt
          20 kB
          Todd Lipcon
        6. hdfs-1172.txt
          21 kB
          Eli Collins
        7. HDFS-1172-150907.patch
          17 kB
          Walter Su
        8. HDFS-1172.008.patch
          14 kB
          Masatake Iwasaki
        9. HDFS-1172.009.patch
          15 kB
          Masatake Iwasaki
        10. HDFS-1172.010.patch
          16 kB
          Masatake Iwasaki
        11. HDFS-1172.011.patch
          16 kB
          Masatake Iwasaki
        12. HDFS-1172.012.patch
          16 kB
          Masatake Iwasaki
        13. HDFS-1172.013.patch
          13 kB
          Masatake Iwasaki
        14. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        15. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki

          Issue Links

            Activity

              People

              • Assignee:
                iwasakims Masatake Iwasaki
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                1 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: