[CASSANDRA-13115] Read repair is not blocking repair to finish in foreground repair

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.11, 3.10

Environment

ccm on OSX

• Severity: Normal
• Assignee: Sylvain Lebresne
• Reporter: Xiaolong Jiang

Authors

Sylvain Lebresne

Reviewers

Jason Brown

• Created: 10/Jan/17 23:00
• Updated: 16/Apr/19 09:30
• Resolved: 19/Jan/17 15:55

Description

The code trying to wait(block) for repair result to come back in 3.X is below:

DataResolver.java

public void close()
        {
            try
            {
                FBUtilities.waitOnFutures(repairResults, DatabaseDescriptor.getWriteRpcTimeout());
            }
            catch (TimeoutException ex)
            {
                // We got all responses, but timed out while repairing
                int blockFor = consistency.blockFor(keyspace);
                if (Tracing.isTracing())
                    Tracing.trace("Timed out while read-repairing after receiving all {} data and digest responses", blockFor);
                else
                    logger.debug("Timeout while read-repairing after receiving all {} data and digest responses", blockFor);

                throw new ReadTimeoutException(consistency, blockFor-1, blockFor, true);
            }
        }

in DataResolver class, but this close method is never called and it's also not auto close(RepairMergeListener is not extending from AutoCloseable/CloseableIterator) which means we never wait for repair to finish before returning final result.

The steps to reproduce:
1. create some keyspace/table with RF = 2
2. start 2 nodes using ccm
3. stop node2
4. disable node1 hinted hand off
5. write some data to node1 with consistency level one
6. start node2
7. query some data from node1
This should trigger read repair. I put some log in above close method, and can not see log print put.

So this bug will basically violate "monotonic quorum reads " guarantee.

Issue Links

• duplicates: CASSANDRA-12368 Background read repair not fixing replicas