CASSANDRA-8346 avoids stale reads during CAS when checking the condition or doing serial reads by disallowing more than one pending endpoint.
It seems like it should be possible to allow more than one pending endpoint by being smarter about who we read from during the QUORUM read or about the state of pending nodes that are there for host replacement.
Well, I guess things are working as they do for decently good reason here. That said, thinking about it, it could be that the solution from
CASSANDRA-8346is a bit of a big hammer: I believe it's enough to ensure that we read from at least one replica that responded to PREPARE 'in the same Paxos round' But we have timeouts on the paxos round, so it could be it is possible to reduce drastically the time we consider a node pending for CAS so that it's not a real problem in practice. Something like having pending node move to a "almost there" state before becoming true replica, and staying in that state for basically the max time of a paxos round, and then Paxos might be able to replace "pending" nodes by those "almost there" for PREPARE.