[HBASE-23359] RS going down with NPE when splitting a region with compaction disabled in branch-1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0, 1.4.13
• Assignee: Toshihiro Suzuki
• Reporter: Toshihiro Suzuki
• Created: 03/Dec/19 13:31
• Updated: 08/Dec/19 20:44
• Resolved: 08/Dec/19 14:00

Description

Trying to backport HBASE-22096 to brach-1, I faced the issue where a RS goes down with NPE when splitting a region with compaction disabled.

The steps to reproduce this issue are as follows:

compaction_switch false
create "test", "cf"
(0...2000).each{|i| put "test", "row#{i}", "cf:col", "val#{i}"}
split "test"

Looking at the regionserver log, I saw the following log:

2019-12-03 22:25:38,611 INFO  [RS:0;10.0.1.11:53504-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of test,,1575379535506.50e322ec68162025e17cddffdc2fb17e.; null
java.lang.NullPointerException
	at org.apache.hadoop.hbase.regionserver.HStore.cancelRequestedCompaction(HStore.java:1834)
	at org.apache.hadoop.hbase.regionserver.CompactSplitThread$Rejection.rejectedExecution(CompactSplitThread.java:656)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:401)
	at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSystemCompaction(CompactSplitThread.java:348)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2111)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2097)
	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.openDaughters(SplitTransactionImpl.java:478)
	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsAfterPONR(SplitTransactionImpl.java:549)
	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:532)
	at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
	at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:153)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-12-03 22:25:38,613 FATAL [RS:0;10.0.1.11:53504-splits-0] regionserver.HRegionServer: ABORTING region server 10.0.1.11,53504,1575379011279: Abort; we got an error after point-of-no-return

Issue Links

• links to: GitHub Pull Request #899