[AURORA-1932] Failure accrual detection mechanism for bad agents

• Type: Task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Scheduler
• Assignee: Jordan Ly
• Reporter: Jordan Ly
• Created: 05/Jun/17 16:47
• Updated: 25/Oct/17 00:15

Description

With the introduction of different OfferManager orderings (see https://reviews.apache.org/r/59480/), we run the risk of repeatedly assigning the same task to a bad agent.

We should develop some sort of 'failure accrual' mechanism where we can track how many times tasks fail on a agent. If it reaches some sort of threshold, we should blacklist that agent for some time so that it can be investigated and the task can be assigned to a different agent.