[MESOS-770] Rate control and randomization of Replicated Log catching-up

• Type: Improvement
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Component/s: replicated log
• Labels: mesosphere
• Assignee: Unassigned
• Reporter: Yan Xu
• Created: 24/Oct/13 21:22
• Updated: 02/Oct/15 15:18

Description

When the log is catching up either in the process of recovering or after coordinator failover the Paxos protocol is run on multiple positions (possibly the entire log).

Currently the catch-up process is linear (one thread fills positions one-by-one). What's preventing us from catching up all positions concurrently is that too much concurrency could have negative impact on the network and the problem may be exacerbated by the contention between multiple recovering replicas and the coordinator.

Rate control helps limit the number of concurrent positions a proposer (recoverer or coordinator) seeks consensus at a time. We can batch a number of positions each time.

Randomly picking the positions in each batch reduces the possibility that multiple proposers contend for the same position at the same time which causes conflict and retries.

Issue Links

• depends upon: MESOS-736 Support catch-up replicated log
• depends upon: MESOS-805 Improve limiter.hpp to allow acquiring bulk permits and releasing permits
• is related to: AURORA-111 Support warm failover in schedulers
• relates to: MESOS-3572 Faster catchup protocol for "learned" replicated log entries