Description
Trying to backport HBASE-22096 to brach-1, I faced the issue where a RS goes down with NPE when splitting a region with compaction disabled.
The steps to reproduce this issue are as follows:
compaction_switch false create "test", "cf" (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val#{i}"} split "test"
Looking at the regionserver log, I saw the following log:
2019-12-03 22:25:38,611 INFO [RS:0;10.0.1.11:53504-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of test,,1575379535506.50e322ec68162025e17cddffdc2fb17e.; null java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.cancelRequestedCompaction(HStore.java:1834) at org.apache.hadoop.hbase.regionserver.CompactSplitThread$Rejection.rejectedExecution(CompactSplitThread.java:656) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:401) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSystemCompaction(CompactSplitThread.java:348) at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2111) at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2097) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.openDaughters(SplitTransactionImpl.java:478) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsAfterPONR(SplitTransactionImpl.java:549) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:532) at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:153) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-12-03 22:25:38,613 FATAL [RS:0;10.0.1.11:53504-splits-0] regionserver.HRegionServer: ABORTING region server 10.0.1.11,53504,1575379011279: Abort; we got an error after point-of-no-return
Attachments
Issue Links
- links to