[FLINK-12863] Race condition between slot offerings and AllocatedSlotReport

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.7.3, 1.8.1, 1.9.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 16/Jun/19 13:59
• Updated: 27/Mar/20 13:51
• Resolved: 21/Jun/19 07:31

Description

With FLINK-11059 we introduced the AllocatedSlotReport which is used by the TaskExecutor to synchronize its internal view on slot allocations with the view of the JobMaster. It seems that there is a race condition between offering slots and receiving the report because the AllocatedSlotReport is sent by the HeartbeatManagerSenderImpl from a separate thread.

Due to that it can happen that we generate an AllocatedSlotReport just before getting new slots offered. Since the report is sent from a different thread, it can then happen that the response to the slot offerings is sent earlier than the AllocatedSlotReport. Consequently, we might receive an outdated slot report on the TaskExecutor causing active slots to be released.

In order to solve the problem I propose to add a fencing token to the AllocatedSlotReport which is being updated whenever we offer new slots to the JobMaster. When we receive the AllocatedSlotReport on the TaskExecutor we compare the current slot report fencing token with the received one and only process the report if they are equal. Otherwise we wait for the next heartbeat to send us an up to date AllocatedSlotReport.

Issue Links

• is caused by: FLINK-11059 JobMaster may continue using an invalid slot if releasing idle slot meet a timeout
• is duplicated by: FLINK-9827 ResourceManager may receive outdate report of slots status from task manager
• is duplicated by: FLINK-9417 Send heartbeat requests from RPC endpoint's main thread
• relates to: FLINK-12895 TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis
• relates to: FLINK-12865 State inconsistency between RM and TM on the slot status
• links to: GitHub Pull Request #8783