Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1172

Blocks in newly completed files are considered under-replicated too quickly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

        Attachments

        1. HDFS-1172.008.patch
          14 kB
          Masatake Iwasaki
        2. HDFS-1172.009.patch
          15 kB
          Masatake Iwasaki
        3. HDFS-1172.010.patch
          16 kB
          Masatake Iwasaki
        4. HDFS-1172.011.patch
          16 kB
          Masatake Iwasaki
        5. HDFS-1172.012.patch
          16 kB
          Masatake Iwasaki
        6. HDFS-1172.013.patch
          13 kB
          Masatake Iwasaki
        7. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        8. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        9. HDFS-1172.patch
          3 kB
          Boris Shkolnik
        10. hdfs-1172.txt
          21 kB
          Eli Collins
        11. hdfs-1172.txt
          20 kB
          Todd Lipcon
        12. HDFS-1172-150907.patch
          17 kB
          Walter Su
        13. replicateBlocksFUC.patch
          4 kB
          Hairong Kuang
        14. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        15. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang

          Issue Links

            Activity

              People

              • Assignee:
                iwasakims Masatake Iwasaki
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                1 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: