[FLINK-12889] Job keeps in FAILING state

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.7.2, 1.8.1, 1.9.0
• Fix Version/s: 1.7.3, 1.8.2, 1.9.0
• Component/s: Runtime / Task
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Fan Xinpu
• Created: 18/Jun/19 13:37
• Updated: 05/Jul/19 08:42
• Resolved: 05/Jul/19 08:42

Description

There is a topology of 3 operator, such as, source, parser, and persist. Occasionally, 5 subtasks of the source encounters exception and turns to failed, at the same time, one subtask of the parser runs into exception and turns to failed too. The jobmaster gets a message of the parser's failed. The jobmaster then try to cancel all the subtask, most of the subtasks of the three operator turns to canceled except the 5 subtasks of the source, because the state of the 5 ones is already FAILED before jobmaster try to cancel it. Then the jobmaster can not reach a final state but keeps in  Failing state meanwhile the subtask of the source kees in canceling state. 
 
The job run on a flink 1.7 cluster on yarn, and there is only one tm with 10 slots.
 
The attached files contains a jm log , tm log and the ui picture.
 
The exception timestamp is about 2019-06-16 13:42:28.

Issue Links

• links to: GitHub Pull Request #8948
• links to: GitHub Pull Request #8950