Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10301

BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.6.1
    • 2.8.0, 2.7.4, 3.0.0-alpha2
    • namenode
    • None

    Description

      When NameNode is busy a DataNode can timeout sending a block report. Then it sends the block report again. Then NameNode while process these two reports at the same time can interleave processing storages from different reports. This screws up the blockReportId field, which makes NameNode think that some storages are zombie. Replicas from zombie storages are immediately removed, causing missing blocks.

      Attachments

        1. zombieStorageLogs.rtf
          32 kB
          Konstantin Shvachko
        2. HDFS-10301.sample.patch
          4 kB
          Daryn Sharp
        3. HDFS-10301.branch-2.patch
          30 kB
          Konstantin Shvachko
        4. HDFS-10301.branch-2.7.patch
          28 kB
          Vinitha Reddy Gankidi
        5. HDFS-10301.branch-2.7.015.patch
          20 kB
          Vinitha Reddy Gankidi
        6. HDFS-10301.branch-2.015.patch
          24 kB
          Konstantin Shvachko
        7. HDFS-10301.015.patch
          26 kB
          Vinitha Reddy Gankidi
        8. HDFS-10301.014.patch
          25 kB
          Vinitha Reddy Gankidi
        9. HDFS-10301.013.patch
          38 kB
          Vinitha Reddy Gankidi
        10. HDFS-10301.012.patch
          32 kB
          Vinitha Reddy Gankidi
        11. HDFS-10301.011.patch
          32 kB
          Vinitha Reddy Gankidi
        12. HDFS-10301.010.patch
          32 kB
          Vinitha Reddy Gankidi
        13. HDFS-10301.01.patch
          16 kB
          Walter Su
        14. HDFS-10301.009.patch
          32 kB
          Vinitha Reddy Gankidi
        15. HDFS-10301.008.patch
          32 kB
          Vinitha Reddy Gankidi
        16. HDFS-10301.007.patch
          32 kB
          Vinitha Reddy Gankidi
        17. HDFS-10301.006.patch
          31 kB
          Vinitha Reddy Gankidi
        18. HDFS-10301.005.patch
          16 kB
          Colin McCabe
        19. HDFS-10301.004.patch
          29 kB
          Vinitha Reddy Gankidi
        20. HDFS-10301.003.patch
          16 kB
          Colin McCabe
        21. HDFS-10301.002.patch
          11 kB
          Colin McCabe

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10953 Remove single-rpc block reports.

          • Major - Major loss of function.
          • Open
          is broken by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7960 The full block report should prune zombie storages even if they're not empty

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              redvine Vinitha Reddy Gankidi
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: