[HBASE-25331] [HBOSS] TreeLockManager should log warnings about lock contention periodically

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: hbase-filesystem-1.0.0-alpha1
• Fix Version/s: hbase-filesystem-1.0.0-alpha2
• Labels: hbase-filesystem
• Assignee: Wellington Chevreuil
• Reporter: Wellington Chevreuil
• Created: 25/Nov/20 17:50
• Updated: 02/Jun/21 15:20
• Resolved: 30/Nov/20 15:10

Description

While performing some internal tests with hboss, psomogyi noticed that TreeLockManager class logs too frequently when it is waiting on parent write lock.

2020-10-26 17:35:41,719 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,720 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,721 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,723 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,724 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,725 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,726 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,728 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,729 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
2020-10-26 17:35:41,730 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001 

In 4 seconds on a single RegionServer this WARN was logged 1767 times.

This modifies current logging logic, to calculate the time spent trying to acquire locks on individual nodes, then warning only given a configurable threshold is exhausted.

Issue Links

• links to: GitHub Pull Request #18