Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10301

BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.1
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None

      Description

      When NameNode is busy a DataNode can timeout sending a block report. Then it sends the block report again. Then NameNode while process these two reports at the same time can interleave processing storages from different reports. This screws up the blockReportId field, which makes NameNode think that some storages are zombie. Replicas from zombie storages are immediately removed, causing missing blocks.

        Attachments

        1. HDFS-10301.002.patch
          11 kB
          Colin P. McCabe
        2. HDFS-10301.003.patch
          16 kB
          Colin P. McCabe
        3. HDFS-10301.004.patch
          29 kB
          Vinitha Reddy Gankidi
        4. HDFS-10301.005.patch
          16 kB
          Colin P. McCabe
        5. HDFS-10301.006.patch
          31 kB
          Vinitha Reddy Gankidi
        6. HDFS-10301.007.patch
          32 kB
          Vinitha Reddy Gankidi
        7. HDFS-10301.008.patch
          32 kB
          Vinitha Reddy Gankidi
        8. HDFS-10301.009.patch
          32 kB
          Vinitha Reddy Gankidi
        9. HDFS-10301.01.patch
          16 kB
          Walter Su
        10. HDFS-10301.010.patch
          32 kB
          Vinitha Reddy Gankidi
        11. HDFS-10301.011.patch
          32 kB
          Vinitha Reddy Gankidi
        12. HDFS-10301.012.patch
          32 kB
          Vinitha Reddy Gankidi
        13. HDFS-10301.013.patch
          38 kB
          Vinitha Reddy Gankidi
        14. HDFS-10301.014.patch
          25 kB
          Vinitha Reddy Gankidi
        15. HDFS-10301.015.patch
          26 kB
          Vinitha Reddy Gankidi
        16. HDFS-10301.branch-2.015.patch
          24 kB
          Konstantin Shvachko
        17. HDFS-10301.branch-2.7.015.patch
          20 kB
          Vinitha Reddy Gankidi
        18. HDFS-10301.branch-2.7.patch
          28 kB
          Vinitha Reddy Gankidi
        19. HDFS-10301.branch-2.patch
          30 kB
          Konstantin Shvachko
        20. HDFS-10301.sample.patch
          4 kB
          Daryn Sharp
        21. zombieStorageLogs.rtf
          32 kB
          Konstantin Shvachko

          Issue Links

            Activity

              People

              • Assignee:
                redvine Vinitha Reddy Gankidi
                Reporter:
                shv Konstantin Shvachko
              • Votes:
                0 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: