Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-763

DataBlockScanner reporting of bad blocks is slightly misleading

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • datanode
    • None
    • Reviewed

    Description

      The Datanode generates a report of the period block scanning that verifies crcs. It reports something like the following:

      Scans since restart : 192266
      Scan errors since restart : 33
      Transient scan errors : 0

      The statement saying that there were 33 errors is slightly midleading because these are not crc mismatches, rather the block was being deleted when the crc verification was about to happen.

      I propose that DataBlockScanner.totalScanErrors is not updated if the dataset.getFile(block) is null, i.e. the block is now deleted from the datanode.

      Attachments

        1. scanErrors.txt
          0.8 kB
          Dhruba Borthakur
        2. scanErrors.txt
          0.8 kB
          Dhruba Borthakur
        3. scanErrors.txt
          2 kB
          Dhruba Borthakur

        Activity

          People

            dhruba Dhruba Borthakur
            dhruba Dhruba Borthakur
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: