[CASSANDRA-9840] global_row_key_cache_test.py fails; loses mutations on cluster restart

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Not A Problem
• Fix Version/s: 3.0.0 rc2
• Component/s: Legacy/Testing
• Severity: Critical
• Since Version: 3.0.x
• Assignee: Ariel Weisberg
• Reporter: Shawn Kumar

Authors

Ariel Weisberg

• Tester: Shawn Kumar
• Created: 17/Jul/15 18:15
• Updated: 16/Apr/19 09:31
• Resolved: 02/Oct/15 16:53

Description

This test is currently failing on trunk. I've attached the test output and logs. It seems that the failure of the test doesn't necessarily have anything to do with global row/key caches - as on the initial loop of the test neither are used and we still hit failure. The test itself fails when a second validation of values after a cluster restart fails to capture deletes issued prior to the restart and first successful validation. However, if I add flushes prior to restarting the cluster the test completes successfully, implying an issue with loss of in-memory mutations due to the cluster restart. Initially I had though this might be due to CASSANDRA-9669, but as Benedict pointed out, the fact that this test has been succeeding consistently on both 2.1 and 2.2 branch indicates there may be another issue at hand.

Issue Links

• relates to: CASSANDRA-9669 If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records