Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19728

Add lock to filesCompacting in all place.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 1.5.0, 2.0.0
    • Fix Version/s: 1.3.2, 1.4.2, 2.0.0, 1.2.7
    • Component/s: None
    • Labels:
      None

      Description

      We find regionserver abort with the following exception:

      2017-05-09 17:40:06,369 FATAL [regionserver/hadoop0349.et2.tbsite.net/11.251.152.199:16020-shortCompactions-1493026663275] regionserver.HRegionServer: ABORTING region server hadoop0349.et2.tbsite.net,16020,1493026637177: Thread[regionserver/hadoop0349.et2.tbsite.net/11.251.152.199:16020-shortCompactions-1493026663275,5,main] throw uncaught exception
      java.lang.ArrayIndexOutOfBoundsException
              at java.lang.System.arraycopy(Native Method)
              at java.util.ArrayList.batchRemove(ArrayList.java:726)
              at java.util.ArrayList.removeAll(ArrayList.java:690)
              at org.apache.hadoop.hbase.regionserver.HStore.finishCompactionRequest(HStore.java:1666)
              at org.apache.hadoop.hbase.regionserver.HStore.cancelRequestedCompaction(HStore.java:1656)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:504)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
              at java.lang.Thread.run(Thread.java:834)

      2017-05-08 21:15:31,979 FATAL [regionserver/hadoop1191.et2.tbsite.net/11.251.159.40:16020-longCompactions-1494249331978] regionserver.HRegionServer: ABORTING region server hadoop1191.et2.tbsite.net,16020,1493196567798: Thread[regionserver/hadoop1191.et2.tbsite.net/11.251.159.40:16020-longCompactions-1494249331978,5,main] throw uncaught exception
      java.lang.IllegalArgumentException
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
              at org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.getCurrentEligibleFiles(RatioBasedCompactionPolicy.java:64)
              at org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.preSelectCompactionForCoprocessor(RatioBasedCompactionPolicy.java:72)
              at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.preSelect(DefaultStoreEngine.java:117)
              at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1542)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:362)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$200(CompactSplitThread.java:58)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:491)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
              at java.lang.Thread.run(Thread.java:834)

      HStore#finishCompactionRequest do not require any HStore#lock's lock so HStore.replaceStoreFiles need to synchronized on filesCompacting.

        Attachments

        1. HBASE-19728.master.001.patch
          0.8 kB
          Lijin Bin
        2. HBASE-19728.master.002.patch
          1 kB
          Lijin Bin
        3. HBASE-19728.master.002.patch
          1 kB
          Chia-Ping Tsai
        4. HBASE-19728.branch-1.001.patch
          1 kB
          Lijin Bin

          Activity

            People

            • Assignee:
              binlijin Lijin Bin
              Reporter:
              binlijin Lijin Bin
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: