[FLINK-16728] Taskmanager dies after job got stuck and canceling fails

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 1.10.0
• Component/s: Runtime / Task
• Assignee: Unassigned
• Reporter: Leonid Ilyevsky
• Created: 23/Mar/20 16:48
• Updated: 25/Sep/20 16:42
• Resolved: 25/Sep/20 16:42

Description

At some point I noticed that a few jobs got stuck (they basically stopped processing the messages, I could detect this watching the expected output), so I tried to cancel them.

The cancel operation failed, complaining that the job got stuck at 

StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:86)

and then the whole taskmanager shut down.

See the attached log.

This is actually happening practically every day in our staging environment where we are testing Flink 1.10.0.