Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
Correctness - Consistency
-
Normal
-
Normal
-
User Report
-
All
-
None
-
Description
For a two DCs cluster setup. When a new node is being added to DC1, for blocking read repair triggered by local_quorum in DC1, it will require to send read repair mutation to an extra node(1)(2). The selector for read repair may select ANY node that has not been contacted before(3) instead of selecting the DC1 nodes. If a node from DC2 is selected, this will cause 100% timeout because of the bug described below:
When we initialized the latch(4) for blocking read repair, the shouldBlockOn function will only return true for local nodes(5), the blockFor value will be reduced if a local node doesn't require repair(6). The blockFor is same as the number of read repair mutation sent out. But when the coordinator node receives the response from the target nodes, the latch only count down for nodes in same DC(7). The latch will wait till timeout and the read request will timeout.
This can be reproduced if you have a constant load on a 3 + 3 cluster when adding a node. If you have someway to trigger blocking read repair(maybe by adding load using stress tool). If you use local_quorum consistency with a constant read after write load in the same DC that you are adding node. You will see read timeout issue from time to time because of the bug described above
I think for read repair when selecting the extra node to do repair, we should prefer local nodes than the nodes from other region. Also, we need to fix the latch part so even if we send mutation to the nodes in other DC, we don't get a timeout.
Attachments
Attachments
Issue Links
- links to
