We have two DC setup for the Cassandra Cluster. 7 nodes each, RF = 3. We setup CL as LOCAL_QUORUM for both read and write.
We have noticed that 95th percentile coordiantorReadLatency is at 60ms range.
We modified read_repair_chance for this CF to 0.0 from the default 0.1, we see dramatic improvements for the coordiantorReadLatency. It went down to 10ms.
I suspect read_repair somehow waits/blocks the response from other DC which shouldn't happen.