Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14849

Erasure Coding: the internal block is replicated many times when datanode is decommissioning

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      colored textWhen the datanode keeping in DECOMMISSION_INPROGRESS status, the EC internal block in that datanode will be replicated many times.

      // added 2019/09/19
      I reproduced this scenario in a 163 nodes cluster with decommission 100 nodes simultaneously.

      Attachments

        1. scheduleReconstruction.png
          124 kB
          HuangTao
        2. liveBlockIndices.png
          450 kB
          HuangTao
        3. HDFS-14849.branch-3.1.patch
          5 kB
          Wei-Chiu Chuang
        4. HDFS-14849.002.patch
          5 kB
          HuangTao
        5. HDFS-14849.001.patch
          5 kB
          HuangTao
        6. fsck-file.png
          2.89 MB
          HuangTao

        Issue Links

        is duplicated by

        Sub-task - The sub-task of the issue HDFS-14450 Erasure Coding: decommissioning datanodes cause replicate a large number of duplicate EC internal blocks

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            marvelrock HuangTao
            marvelrock HuangTao
            Votes:
            0 Vote for this issue
            Watchers:
            8 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment