[MESOS-9330] Default executor can skip status updates

• Type: Bug
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Component/s: executor
• Labels: default-executor foundations mesosphere
• Epic Link: Default Executor improvements
• Assignee: Unassigned
• Reporter: Gastón Kleiman
• Created: 17/Oct/18 20:12
• Updated: 06/Feb/19 17:10

Description

The default executor only retries unacknowledged status updates upon re-registration. However an ERROR event doesn't trigger a re-registration, so it is possible for it to skip status updates, e.g., if there was a transient auth error when it sent the UPDATE event:

I1008 12:46:12.750519     5 checker_process.cpp:826] Output of the COMMAND check for task 'kafka-1-broker__c1201c76-301d-4d1d-ab3b-ee6ddf21db47' (stdout):
Checking for started log line in confluent-4.1.1/logs/server.log*.
Found started log line.
Required log line found. Broker is ready.
I1008 12:46:12.750555     5 checker_process.cpp:829] Output of the COMMAND check for task 'kafka-1-broker__c1201c76-301d-4d1d-ab3b-ee6ddf21db47' (stderr):
I1008 12:46:12.815301     6 checker_process.cpp:986] COMMAND check for task 'kafka-1-broker__c1201c76-301d-4d1d-ab3b-ee6ddf21db47' returned: 0
I1008 12:46:12.815397     6 default_executor.cpp:1327] Received check update 'COMMAND exit code 0' for task 'kafka-1-broker__c1201c76-301d-4d1d-ab3b-ee6ddf21db47'
I1008 12:46:12.858417     6 default_executor.cpp:204] Received ERROR event
E1008 12:46:12.858449     6 default_executor.cpp:294] Error: Received unexpected '401 Unauthorized' ("Bearer" authenticator returned:
Invalid JWT token: Token signature does not match.) for UPDATE