Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23359

RS going down with NPE when splitting a region with compaction disabled in branch-1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0, 1.4.13
    • Component/s: None
    • Labels:
      None

      Description

      Trying to backport HBASE-22096 to brach-1, I faced the issue where a RS goes down with NPE when splitting a region with compaction disabled.

      The steps to reproduce this issue are as follows:

      compaction_switch false
      create "test", "cf"
      (0...2000).each{|i| put "test", "row#{i}", "cf:col", "val#{i}"}
      split "test"
      

      Looking at the regionserver log, I saw the following log:

      2019-12-03 22:25:38,611 INFO  [RS:0;10.0.1.11:53504-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of test,,1575379535506.50e322ec68162025e17cddffdc2fb17e.; null
      java.lang.NullPointerException
      	at org.apache.hadoop.hbase.regionserver.HStore.cancelRequestedCompaction(HStore.java:1834)
      	at org.apache.hadoop.hbase.regionserver.CompactSplitThread$Rejection.rejectedExecution(CompactSplitThread.java:656)
      	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
      	at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:401)
      	at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSystemCompaction(CompactSplitThread.java:348)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2111)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2097)
      	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.openDaughters(SplitTransactionImpl.java:478)
      	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsAfterPONR(SplitTransactionImpl.java:549)
      	at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:532)
      	at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
      	at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:153)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      2019-12-03 22:25:38,613 FATAL [RS:0;10.0.1.11:53504-splits-0] regionserver.HRegionServer: ABORTING region server 10.0.1.11,53504,1575379011279: Abort; we got an error after point-of-no-return
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brfrn169 Toshihiro Suzuki
                Reporter:
                brfrn169 Toshihiro Suzuki
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: