[HBASE-23968] Periodically check whether a system stop is requested in compaction by time.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1
• Component/s: Compaction
• Hadoop Flags: Reviewed

Release Note

All compactions must be stopped before the region server is stopped. If compaction is running, the region server wait for the compaction is stopped. Previously, The compaction check that the system is stopped when it reaches "hbase.hstore.close.check.interval" (size). However, depending on the situation, it may take a long time to reach that size. This could result in delays in the stop of the region server. To solve this problem, "hbase.hstore.close.check.time.interval" was added. Now, The compaction check that the system is stopped when it reaches "hbase.hstore.close.check.interval" (size) or "hbase.hstore.close.check.time.interval" (time). The default value of "hbase.hstore.close.check.time.interval" is 10 seconds.

• Assignee: Minwoo Kang
• Reporter: Minwoo Kang
• Created: 11/Mar/20 10:14
• Updated: 19/Aug/21 18:17
• Resolved: 04/May/20 21:48

Description

The compaction check that the system is stopped when it reaches hbase.hstore.close.check.interval size.
If the size is not reached, the compaction does not stop even though the system is stopped.
If compaction is not terminated, the region is not closed.
As a result, region remains pending-close.
 
Thread A (regionserver/longCompactions):
State: RUNNABLE
Stack:
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:319)
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:111)
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119)
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1232)
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1896)
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:525)
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:562)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)

Thread B (RS_CLOSE_REGION):
State: WAITING
Waiting on org.apache.hadoop.hbase.regionserver.HRegion$WriteState
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1567)
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1417)
org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1380)
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
 

Issue Links

• links to: GitHub Pull Request #1274

Sub-Tasks

• Backport HBASE-23968 to branch-2: Resolved
• Backport HBASE-23968 to branch-1: Resolved
• CloseChecker should think Pressure Aware Throughput Controller: Open