Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
3.11.16, 4.0.11, 4.1.3, 5.0-alpha1, 5.0
-
None
-
Availability
-
Normal
-
Normal
-
User Report
-
All
-
None
-
Description
When a node starts it will get endpoint states (via shadow round) but have all nodes marked as down. The problem is the wait to settle only checks the size of endpoint states is stable before starting Native transport. Once native transport starts it will receive queries and fail consistency levels such as LOCAL_QUORUM since it still thinks nodes are down.
This is problem for a number of large clusters for our customers. The cluster has quorum but due to this issue a node restart is causing a bunch of query errors.
My initial solution to this was to only check live endpoints size in addition to size of endpoint states. This worked but I noticed in testing this fix that there also a lot of duplication of checking the same node (via Echo messages) for liveness. So the patch also removes this duplication of checking node is UP in markAlive.
The final problem I found while testing is sometimes could still not see a change in live endpoints due to only 1 second polling, so the patch allows for overridding the settle parameters. I could not reliability reproduce this but think its worth providing a way to override these hardcoded values.