[HBASE-18158] Two running in-memory compaction threads may lose data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.0.0
• Fix Version/s: 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Chia-Ping Tsai
• Reporter: Chia-Ping Tsai
• Created: 04/Jun/17 15:48
• Updated: 01/Aug/18 06:22
• Resolved: 07/Jun/17 10:05

Description

CompactingMemStore.java

  private void stopCompaction() {
    if (inMemoryFlushInProgress.get()) {
      compactor.stop();
      inMemoryFlushInProgress.set(false);
    }
  }

The stopCompaction() set inMemoryFlushInProgress to false so there may be two in-memory compaction threads which execute simultaneously. If there are two running InMemoryFlushRunnable, the later InMemoryFlushRunnable may change the versionedList.

MemStoreCompactor.java

  public boolean start() throws IOException {
    if (!compactingMemStore.hasImmutableSegments()) { // no compaction on empty pipeline
      return false;
    }

    // get a snapshot of the list of the segments from the pipeline,
    // this local copy of the list is marked with specific version
    versionedList = compactingMemStore.getImmutableSegments();

And the first InMemoryFlushRunnable will use the chagned versionedList to remove the corresponding segments.

MemStoreCompactor#doCompaction

      if (!isInterrupted.get()) {
        if (resultSwapped = compactingMemStore.swapCompactedSegments(
            versionedList, result, (action==Action.MERGE))) {
          // update the wal so it can be truncated and not get too long
          compactingMemStore.updateLowestUnflushedSequenceIdInWAL(true); // only if greater
        }
      }

In conclusion, first InMemoryFlushRunnable will remove the worng segment. And the later InMemoryFlushRunnable may introduce NPE because first InMemoryFlushRunnable set versionedList to null after compaction.

Exception in thread "RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=45712-inmemoryCompactions-1496563908038" java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:119)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:283)
        at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:212)
        at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:122)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:388)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:500)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

Issue Links

• links to: RB