Details
-
Improvement
-
Status: Resolved
-
Critical
-
Resolution: Duplicate
-
None
-
None
Description
Per some discussions with tweingartner and vinodkone.
Currently the master uses a SlaveObserver for each registered slave. Each SlaveObserver operates independently and makes decisions about whether the slave is healthy.
The independence of these observers means that in some very rare events (e.g. masters are partitioned from 75% of slaves), the master can very rapidly remove a large portion of the slaves in the cluster. Ideally such an event could be deemed dangerous and throttled accordingly through a more intelligent notion of overall cluster health.
It may be nice to have a single observer that is responsible for health checking all the slaves. This will allow us to make safer decisions as to when to determine that slaves are unhealthy.