[HBASE-2856] TestAcidGuarantee broken on trunk

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.89.20100621
• Fix Version/s: 0.94.0
• Assignee: Amitanand Aiyer
• Reporter: ryan rawson
• Created: 20/Jul/10 19:28
• Updated: 12/Oct/12 05:35
• Resolved: 18/Nov/11 02:21

Description

TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk.

One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas).

Issue Links

• blocks: HBASE-3149 Make flush decisions per column family
• is related to: HBASE-5121 MajorCompaction may affect scan's correctness
• is related to: HBASE-4570 Scan ACID problem with concurrent puts.
• is required by: HBASE-4838 Port 2856 (TestAcidGuarantee is failing) to 0.92
• is required by: HBASE-3404 Compaction Ordering for Bulk Import Files
• is required by: HBASE-3290 Max Compaction Size
• relates to: HBASE-3498 Memstore scanner needs new semantics, which may require new data structure
• relates to: HBASE-5569 Do not collect deleted KVs when they are still in use by a scanner.
• relates to: HBASE-3543 Get rid of "delete forward"

Sub-Tasks

• Memstore scanner needs new semantics, which may require new data structure: Closed
• Get the TestAcidGuarantee unit test to fail consistently: Closed
• Persist memstoreTS to disk: Closed
• Ensure that Scanners that read from the storefiles respect MVCC: Closed
• Optimise the storage that we use for storing MVCC information.: Closed
• Use variable length format to store the memstoreTS: Closed
• Eliminate window of missing Data: Closed
• Rename RWCC to MVCC: Closed
• Ensure that KV's newer than the oldest-living-scanner is not accounted for the maxVersions during flush/compaction.: Closed
• Port 2856 (TestAcidGuarantee is failing) to 0.92: Closed
• Make sure we clean up scannerReadPoints upon any exceptions: Closed