Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6147

New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      New blocks scanning will be delayed if old blocks deleted after datanode restart.

      Steps:
      1. Write some blocks and wait till all scans over
      2. Restart the datanode
      3. Delete some of the blocks
      4. Write new blocks which are less in size compared to deleted blocks.

      Problem:
      BlockPoolSliceScanner#updateBytesToScan(..) updates bytesLeft based on following comparison

         if (lastScanTime < currentPeriodStart) {
            bytesLeft += len;
          }

      But in BlockPoolSliceScanner#assignInitialVerificationTimes() bytesLeft decremented using below comparison

      if (now - entry.verificationTime < scanPeriod) {

      Hence when the old blocks are deleted bytesLeft going negative.
      new blocks will not be scanned until it becomes positive again.

      So in both places verificationtime should be compared against scanperiod.

      1. HDFS-6147.patch
        8 kB
        Vinayakumar B
      2. HDFS-6147.patch
        13 kB
        Vinayakumar B
      3. HDFS-6147.patch
        13 kB
        Vinayakumar B
      4. HDFS-6147.patch
        13 kB
        Vinayakumar B
      5. HDFS-6147.patch
        14 kB
        Vinayakumar B
      6. HDFS-6147.patch
        14 kB
        Vinayakumar B

        Issue Links

          Activity

          Vinayakumar B created issue -
          Vinayakumar B made changes -
          Field Original Value New Value
          Attachment HDFS-6147.patch [ 12636312 ]
          Vinayakumar B made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinayakumar B made changes -
          Attachment HDFS-6147.patch [ 12636865 ]
          Vinayakumar B made changes -
          Attachment HDFS-6147.patch [ 12641447 ]
          Vinayakumar B made changes -
          Attachment HDFS-6147.patch [ 12645955 ]
          Vinayakumar B made changes -
          Attachment HDFS-6147.patch [ 12657553 ]
          Vinayakumar B made changes -
          Attachment HDFS-6147.patch [ 12666666 ]
          Vinayakumar B made changes -
          Link This issue is part of HDFS-7430 [ HDFS-7430 ]
          Vinayakumar B made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]

            People

            • Assignee:
              Vinayakumar B
              Reporter:
              Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development