Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12914

Block report leases cause missing blocks until next report

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.9.2
    • Fix Version/s: 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None

      Description

      BlockReportLeaseManager#checkLease will reject FBRs from DNs for conditions such as "unknown datanode", "not in pending set", "lease has expired", wrong lease id, etc. Lease rejection does not throw an exception. It returns false which bubbles up to NameNodeRpcServer#blockReport and interpreted as noStaleStorages.

      A re-registering node whose FBR is rejected from an invalid lease becomes active with no blocks. A replication storm ensues possibly causing DNs to temporarily go dead (HDFS-12645), leading to more FBR lease rejections on re-registration. The cluster will have many "missing blocks" until the DNs next FBR is sent and/or forced.

        Attachments

        1. HDFS-12914-trunk.00.patch
          4 kB
          Santosh Marella
        2. HDFS-12914-branch-2.001.patch
          5 kB
          Santosh Marella
        3. HDFS-12914-trunk.01.patch
          4 kB
          Santosh Marella
        4. HDFS-12914.005.patch
          11 kB
          Xiaoqiao He
        5. HDFS-12914.006.patch
          11 kB
          Xiaoqiao He
        6. HDFS-12914.007.patch
          11 kB
          Xiaoqiao He
        7. HDFS-12914.008.patch
          11 kB
          Xiaoqiao He
        8. HDFS-12914.branch-3.2.patch
          13 kB
          Wei-Chiu Chuang
        9. HDFS-12914.009.patch
          12 kB
          Xiaoqiao He
        10. HDFS-12914.utfix.patch
          2 kB
          Xiaoqiao He
        11. HDFS-12914.branch-3.1.001.patch
          8 kB
          Xiaoqiao He
        12. HDFS-12914.branch-3.1.002.patch
          13 kB
          Xiaoqiao He
        13. HDFS-12914.branch-2.patch
          13 kB
          Wei-Chiu Chuang
        14. HDFS-12914.branch-3.0.patch
          13 kB
          Xiaoqiao He
        15. HDFS-12914.branch-2.000.patch
          13 kB
          Wei-Chiu Chuang
        16. HDFS-12914.branch-2.001.patch
          13 kB
          Wei-Chiu Chuang
        17. HDFS-12914.branch-2.002.patch
          13 kB
          Wei-Chiu Chuang
        18. HDFS-12914.branch-2.8.001.patch
          13 kB
          Wei-Chiu Chuang
        19. HDFS-12914.branch-2.8.002.patch
          13 kB
          Wei-Chiu Chuang

          Issue Links

            Activity

              People

              • Assignee:
                smarella Santosh Marella
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: