[FLINK-12865] State inconsistency between RM and TM on the slot status

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.3, 1.8.1, 1.9.0
• Component/s: Runtime / Coordination
• Assignee: Till Rohrmann
• Reporter: Yun Gao
• Created: 17/Jun/19 03:49
• Updated: 21/Jun/19 07:31
• Resolved: 21/Jun/19 07:31

Description

There may be state inconsistency between TM and RM due to race condition and message loss:

1. When TM sends heartbeat, it retrieve SlotReport in the main thread, but sends the heartbeat in another thread. There may be cases that the slot on TM is FREE initially and SlotReport read the FREE state, then RM requests slot and mark the slot as allocated, and the SlotReport finally override the allocated status at the RM side wrongly.
2. When RM requests slot, TM received the requests but the acknowledge message get lot. Then RM will think this slot is free. 

 Both the problems may cause RM marks an ALLOCATED slot as FREE. This may currently cause additional retries till the state is synchronized after the next heartbeat, and for the inaccurate resource statistics for the fine-grained resource management in the future.

Issue Links

• is related to: FLINK-12863 Race condition between slot offerings and AllocatedSlotReport
• links to: Github Pull Request #8783