Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6147

New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      New blocks scanning will be delayed if old blocks deleted after datanode restart.

      Steps:
      1. Write some blocks and wait till all scans over
      2. Restart the datanode
      3. Delete some of the blocks
      4. Write new blocks which are less in size compared to deleted blocks.

      Problem:
      BlockPoolSliceScanner#updateBytesToScan(..) updates bytesLeft based on following comparison

         if (lastScanTime < currentPeriodStart) {
            bytesLeft += len;
          }

      But in BlockPoolSliceScanner#assignInitialVerificationTimes() bytesLeft decremented using below comparison

      if (now - entry.verificationTime < scanPeriod) {

      Hence when the old blocks are deleted bytesLeft going negative.
      new blocks will not be scanned until it becomes positive again.

      So in both places verificationtime should be compared against scanperiod.

        Attachments

        1. HDFS-6147.patch
          14 kB
          Vinayakumar B
        2. HDFS-6147.patch
          14 kB
          Vinayakumar B
        3. HDFS-6147.patch
          13 kB
          Vinayakumar B
        4. HDFS-6147.patch
          13 kB
          Vinayakumar B
        5. HDFS-6147.patch
          13 kB
          Vinayakumar B
        6. HDFS-6147.patch
          8 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayrpet Vinayakumar B
                Reporter:
                vinayrpet Vinayakumar B
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: