[AURORA-1719] Aurora warns about every terminated executor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.16.0
• Assignee: Stephan Erb
• Reporter: Stephan Erb
• Created: 16/Jun/16 15:13
• Updated: 19/Sep/16 18:49
• Resolved: 23/Aug/16 16:18

Description

A clean executor shutdown in Mesos:

I0616 13:55:16.580080 16915 master.cpp:4891] Executor 'thermos-role-env-job-0-d94972f8-760e-4bb0-beef-654e2df1f5e0' of framework 20151001-085346-58917130-5050-37976-0000 on slave d4218d85-e294-4405-af4c-80fc7a66f1a4
-S0 at slave(1)@xxxx:5051 (xxxx): exited with status 0
I0616 13:55:16.580286 16915 master.cpp:6540] Removing executor 'thermos-role-env-job-0-d94972f8-760e-4bb0-beef-654e2df1f5e0' with resources cpus(*):0.01; mem(*):128 of framework 20151001-085346-58917130-5050-37976-0000 on slave d4218d85-e294-4405-af4c-80fc7a66f1a4-S0 at slave(1)@xxxx:5051 (xxxx)

On the Aurora end, this leads to this warning and stats increment of scheduler_lost_executors:

W0616 14:17:14.603 [Thread-7189, MesosSchedulerImpl:258] Lost executor value: "thermos-role-env-job-0-d94972f8-760e-4bb0-beef-654e2df1f5e0"

I read the responsible mesos commit as it is fully intentional to always send the executorLost even if the shutdown was clean . We should therefore add a guard to our handler to only dispatch a warning if necessary.

Issue Links

• depends upon: MESOS-313 Report executor terminations to framework schedulers.