[HBASE-18099] FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0, 2.0.0, 1.2.7
• Component/s: snapshots
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 24/May/17 15:37
• Updated: 07/Sep/18 01:57
• Resolved: 25/May/17 14:57

Description

In the following thread:
http://search-hadoop.com/m/HBase/YGbbMXkeHlI9zo
Jacob described the scenario where data from certain region were missing in the snapshot.

Here was related region server log:
https://pastebin.com/1ECXjhRp

He pointed out that concurrent flush from MemStoreFlusher.1 thread was not initiated from the thread pool for snapshot.

In RegionSnapshotTask#call() method there is this:

          region.flush(true);

The return value is not checked.

In HRegion#flushcache(), Result.CANNOT_FLUSH may be returned due to:

          String msg = "Not flushing since "
              + (writestate.flushing ? "already flushing"
              : "writes not enabled");

This implies that FlushSnapshotSubprocedure may incorrectly skip waiting for the concurrent flush to complete.

Issue Links

• is related to: HBASE-18358 Backport HBASE-18099 'FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish' to branch-1.3