[CASSANDRA-6887] LOCAL_ONE read repair only does local repair, in spite of global digest queries

• Type: Bug
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Fix Version/s: 3.11.x
• Labels: repair

Environment

Cassandra 2.0.6, x86-64 ubuntu precise

• Severity: Low
• Assignee: Unassigned
• Reporter: Duncan Sands
• Created: 19/Mar/14 09:13
• Updated: 16/Apr/19 09:31

Description

I have a cluster spanning two data centres. Almost all of the writing (and a lot of reading) is done in DC1. DC2 is used for running the occasional analytics query. Reads in both data centres use LOCAL_ONE. Read repair settings are set to the defaults on all column families.

I had a long network outage between the data centres; it lasted longer than the hints window, so after it was over DC2 didn't have the latest information. Even after reading data many many times in DC2, the returned data was still out of date: read repair was not correcting it.

I then investigated using cqlsh in DC2, with tracing on.

What I saw was:

• with consistency ONE, after about 10 read requests a digest request would be sent to many nodes (spanning both data centres), and the data in DC2 would be repaired.

• with consistency LOCAL_ONE, after about 10 read requests a digest request would be sent to many nodes (spanning both data centres), but the data in DC2 would not be repaired. This is in spite of digest requests being sent to DC1, as shown by the tracing.

So it looks like digest requests are being sent to both data centres, but replies from outside the local data centre are ignored when using LOCAL_ONE.

The same data is being queried all the time in DC1 with consistency LOCAL_ONE, but this didn't result in the data in DC2 being read repaired either. This is a slightly different case to what I described above: in that case the local node was out of date and the remote node had the latest data, while here it is the other way round.

It could be argued that you don't want cross data centre read repair when using LOCAL_ONE. But then why bother sending cross data centre digest requests? And if only doing local read repair is how it is supposed to work then it would be good to document this somewhere.

Issue Links

• is duplicated by: CASSANDRA-7633 Speculating retry for LOCAL_QUORUM send requests to other DC
• relates to: CASSANDRA-9753 LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch
• relates to: CASSANDRA-7902 Introduce the ability to ignore RR based on consistency