[HDFS-6147] New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: datanode
• Assignee: Vinayakumar B
• Reporter: Vinayakumar B
• Created: 24/Mar/14 08:51
• Updated: 09/Mar/15 08:41
• Resolved: 09/Mar/15 08:41

Description

New blocks scanning will be delayed if old blocks deleted after datanode restart.

Steps:
1. Write some blocks and wait till all scans over
2. Restart the datanode
3. Delete some of the blocks
4. Write new blocks which are less in size compared to deleted blocks.

Problem:
BlockPoolSliceScanner#updateBytesToScan(..) updates bytesLeft based on following comparison

   if (lastScanTime < currentPeriodStart) {
      bytesLeft += len;
    }

But in BlockPoolSliceScanner#assignInitialVerificationTimes() bytesLeft decremented using below comparison

if (now - entry.verificationTime < scanPeriod) {

Hence when the old blocks are deleted bytesLeft going negative.
new blocks will not be scanned until it becomes positive again.

So in both places verificationtime should be compared against scanperiod.

Issue Links

• is part of: HDFS-7430 Rewrite the BlockScanner to use O(1) memory and use multiple threads