[AURORA-1459] DelayExecutor is flaky within scheduling loop

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Scheduler
• Sprint: Twitter Aurora Q3'15 Sprint 11
• Story Points: 5
• Assignee: Bill Farner
• Reporter: Maxim Khutornenko
• Created: 29/Aug/15 20:07
• Updated: 10/Nov/15 17:59
• Resolved: 15/Sep/15 01:29

Description

TaskGroups now uses DelayExecutor introduced to gate async operations. The problem though is that DelayExecutor queue is only flushed on DB transaction completion (1). This means no scheduling can ever proceed unless there is some storage mutation activity. If/when there are no storage writes scheduling effectively halts.

While it unlikely to happen in production, it is consistently reproducible with e2e tests in vagrant on any subsequent run.

(1) - https://github.com/apache/aurora/blob/06ddaadbcba4c66b8019815de6ca27d50a9df77d/src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java#L175-L178