[HBASE-11263] Share the open/close store file thread pool for all store in a region

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Affects Version/s: 0.99.0
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Shaohui Liu
• Created: 28/May/14 13:53
• Updated: 17/Jun/22 15:58
• Resolved: 17/Jun/22 15:58

Description

Currently, the open/close store file thread pool is divided equally to all stores of a region.

  protected ThreadPoolExecutor getStoreFileOpenAndCloseThreadPool(
      final String threadNamePrefix) {
    int numStores = Math.max(1, this.htableDescriptor.getFamilies().size());
    int maxThreads = Math.max(1,
        conf.getInt(HConstants.HSTORE_OPEN_AND_CLOSE_THREADS_MAX,
            HConstants.DEFAULT_HSTORE_OPEN_AND_CLOSE_THREADS_MAX)
            / numStores);
    return getOpenAndCloseThreadPool(maxThreads, threadNamePrefix);
  }

This is not very optimal in following scenarios:

1. The data of some column families are very large and there are many hfiles in those stores, and others may be very small and in-memory column families.
2. Usually we preserve some column families for later needs. The thread pool for these column families are wasted。

The simple way is to share a big thread pool for all stores to open/close hfiles.

Suggestions are welcomed.
Thanks.