Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25331

[HBOSS] TreeLockManager should log warnings about lock contention periodically

    XMLWordPrintableJSON

    Details

      Description

      While performing some internal tests with hboss, Peter Somogyi noticed that TreeLockManager class logs too frequently when it is waiting on parent write lock.

      2020-10-26 17:35:41,719 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,720 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,721 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,723 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,724 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,725 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,726 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,728 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,729 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
      2020-10-26 17:35:41,730 WARN org.apache.hadoop.hbase.oss.sync.TreeLockManager: Blocked waiting for some parent write lock, waiting: /s3a/hbase/hbase-dl/db1/hbase/data/default/test1/.tabledesc/.tableinfo.0000000001 

      In 4 seconds on a single RegionServer this WARN was logged 1767 times.

      This modifies current logging logic, to calculate the time spent trying to acquire locks on individual nodes, then warning only given a configurable threshold is exhausted.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wchevreuil Wellington Chevreuil
                Reporter:
                wchevreuil Wellington Chevreuil
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: