Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15406

Improve the speed of Datanode Block Scan

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.2, 3.3.1, 3.4.0
    • None
    • None

    Description

      In our customer cluster we have approx 10M blocks in one datanode 

      the Datanode to scans all the blocks , it has taken nearly 5mins

      2020-06-10 12:17:06,869 | INFO  | java.util.concurrent.ThreadPoolExecutor$Worker@3b4bea70[State = -1, empty queue] | BlockPool BP-1104115233-**.**.**.**-1571300215588 Total blocks: 11149530, missing metadata files:472, missing block files:472, missing blocks in memory:0, mismatched blocks:0 | DirectoryScanner.java:473
      2020-06-10 12:17:06,869 | WARN  | java.util.concurrent.ThreadPoolExecutor$Worker@3b4bea70[State = -1, empty queue] | Lock held time above threshold: lock identifier: org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl lockHeldTimeMs=329854 ms. Suppressed 0 lock warnings. The stack trace is: java.lang.Thread.getStackTrace(Thread.java:1559)
      org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
      org.apache.hadoop.util.InstrumentedLock.logWarning(InstrumentedLock.java:148)
      org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:186)
      org.apache.hadoop.util.InstrumentedLock.unlock(InstrumentedLock.java:133)
      org.apache.hadoop.util.AutoCloseableLock.release(AutoCloseableLock.java:84)
      org.apache.hadoop.util.AutoCloseableLock.close(AutoCloseableLock.java:96)
      org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:475)
      org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
      org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)
       | InstrumentedLock.java:143 

      Attachments

        1. HDFS-15406.001.patch
          1 kB
          Hemanth Boyina
        2. HDFS-15406.002.patch
          2 kB
          Hemanth Boyina

        Issue Links

          Activity

            People

              hemanthboyina Hemanth Boyina
              hemanthboyina Hemanth Boyina
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: