Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1172

Blocks in newly completed files are considered under-replicated too quickly

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 2.8.0, 3.0.0-alpha1
    • namenode
    • None
    • Reviewed

    Description

      I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

      Attachments

        1. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        2. replicateBlocksFUC1.patch
          8 kB
          Hairong Kuang
        3. replicateBlocksFUC.patch
          4 kB
          Hairong Kuang
        4. HDFS-1172-150907.patch
          17 kB
          Walter Su
        5. hdfs-1172.txt
          20 kB
          Todd Lipcon
        6. hdfs-1172.txt
          21 kB
          Eli Collins
        7. HDFS-1172.patch
          3 kB
          Boris Shkolnik
        8. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        9. HDFS-1172.014.patch
          13 kB
          Masatake Iwasaki
        10. HDFS-1172.013.patch
          13 kB
          Masatake Iwasaki
        11. HDFS-1172.012.patch
          16 kB
          Masatake Iwasaki
        12. HDFS-1172.011.patch
          16 kB
          Masatake Iwasaki
        13. HDFS-1172.010.patch
          16 kB
          Masatake Iwasaki
        14. HDFS-1172.009.patch
          15 kB
          Masatake Iwasaki
        15. HDFS-1172.008.patch
          14 kB
          Masatake Iwasaki

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            iwasakims Masatake Iwasaki
            tlipcon Todd Lipcon
            Votes:
            1 Vote for this issue
            Watchers:
            33 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment