[HIVE-26495] MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Labels: pull-request-available
• Assignee: Naresh P R
• Reporter: Naresh P R
• Created: 24/Aug/22 19:26
• Updated: 16/Nov/22 13:50
• Resolved: 26/Sep/22 05:13

Description

With hive.metastore.fshandler.threads = 15, all 15 MSCK-GetPaths-xx are slogging at following trace.

"MSCK-GetPaths-11" #12345 daemon prio=5 os_prio=0 tid= nid= waiting on condition [0x00007f9f099a6000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000003f92d1668> (a java.util.concurrent.CompletableFuture$Signaller)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
    at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
...
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3230)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1995)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.processPathDepthInfo(HiveMetaStoreChecker.java:550)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:543)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:525)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

We should take advantage of non-block listStatusIterator instead of listStatus which is a blocking call.

Issue Links

• links to: GitHub Pull Request #3549