[AURORA-1933] Scheduler can process rescind before offer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.18.0
• Assignee: Zameer Manji
• Reporter: Zameer Manji
• Created: 06/Jun/17 01:07
• Updated: 22/Aug/17 00:38
• Resolved: 06/Jun/17 21:44

Description

I observed the following in production:

Jun  6 00:31:32 compute1159-dca1 aurora-scheduler[23675]: I0606 00:31:32.510 [Thread-77638, MesosCallbackHandler$MesosCallbackHandlerImpl:229] Offer rescinded: 81e04cbd-9bce-41cf-bd94-38c911f255e4-O142359552
Jun  6 00:31:32 compute1159-dca1 aurora-scheduler[23675]: I0606 00:31:32.903 [SchedulerImpl-0, MesosCallbackHandler$MesosCallbackHandlerImpl:211] Received offer: 81e04cbd-9bce-41cf-bd94-38c911f255e4-O142359552
Jun  6 00:31:34 compute1159-dca1 aurora-scheduler[23675]: I0606 00:31:34.815 [TaskGroupBatchWorker, VersionedSchedulerDriverService:123] Accepting offer 81e04cbd-9bce-41cf-bd94-38c911f255e4-O142359552 with ops [LAUNCH]

Notice the rescind was processed before the offer was given. This means the offer is in the offer storage, but using it is invalid. It will cause whatever task launched with it to fail with Task launched with invalid offers: Offer 81e04cbd-9bce-41cf-bd94-38c911f255e4-O142359552 is no longer valid

Issue Links

• is related to: AURORA-1945 Rescinds received but not processed in time before offer accept