[MESOS-9022] Race condition in task updates could cause missing event in streaming

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: HTTP API, master
• Labels: events foundations mesos mesosphere race-condition streaming
• Flags: Important
• Assignee: Benno Evers
• Reporter: Evelyn Liu
• Shepherd: Zhitao Li
• Created: 22/Jun/18 23:56
• Updated: 03/Dec/18 18:38
• Resolved: 03/Dec/18 15:55

Description

Master sends update event of TASK_STARTING when task's latest state is already TASK_FAILED. Then when it handles the update of TASK_FAILED, sendSubscribersUpdate is set to false because of this. The subscriber would not receive update event of TASK_FAILED.

This happened when a task failed very fast. Is there a race condition while handling task updates?

master log:

I0622 13:08:29.189771 84079 master.cpp:8345] Status update TASK_STARTING (Status UUID: eb091093-d303-4e82-b69f-e2ba1011ba76) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 from agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587
 I0622 13:08:29.189801 84079 master.cpp:8402] Forwarding status update TASK_STARTING (Status UUID: eb091093-d303-4e82-b69f-e2ba1011ba76) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000
 I0622 13:08:29.190004 84079 master.cpp:10843] Updating the state of task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (latest state: TASK_STARTING, status update state: TASK_STARTING)
 I0622 13:08:29.603857 84079 master.cpp:6195] Processing ACKNOWLEDGE call for status eb091093-d303-4e82-b69f-e2ba1011ba76 for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (Aurora) on agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587
 I0622 13:08:29.615643 84079 master.cpp:8345] Status update TASK_STARTING (Status UUID: eb091093-d303-4e82-b69f-e2ba1011ba76) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 from agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587
 I0622 13:08:29.615669 84079 master.cpp:8402] Forwarding status update TASK_STARTING (Status UUID: eb091093-d303-4e82-b69f-e2ba1011ba76) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000
 I0622 13:08:29.615783 84079 master.cpp:10843] Updating the state of task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (latest state: TASK_FAILED, status update state: TASK_STARTING)
 I0622 13:08:29.620837 84079 master.cpp:8345] Status update TASK_FAILED (Status UUID: ac34f1e9-eaa4-4765-82ac-7398c2e6c835) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 from agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587
 I0622 13:08:29.620853 84079 master.cpp:8402] Forwarding status update TASK_FAILED (Status UUID: ac34f1e9-eaa4-4765-82ac-7398c2e6c835) for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000
 I0622 13:08:29.620923 84079 master.cpp:10843] Updating the state of task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (latest state: TASK_FAILED, status update state: TASK_FAILED)
 I0622 13:08:29.630455 84079 master.cpp:6195] Processing ACKNOWLEDGE call for status eb091093-d303-4e82-b69f-e2ba1011ba76 for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (Aurora) on agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587
 I0622 13:08:29.673051 84095 master.cpp:6195] Processing ACKNOWLEDGE call for status ac34f1e9-eaa4-4765-82ac-7398c2e6c835 for task f839055c-7a40-4e6c-9f53-22030f388c8c of framework 4591ea8b-4adb-4acf-bb29-b70817663c4e-0000 (Aurora) on agent d2f1c7c2-668d-46e5-829b-ce614cca79ae-S1587

 

Issue Links

• is related to: MESOS-9000 Operator API event stream can miss task status updates.