[FLINK-8020] Deadlock found in Async I/O operator

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 1.3.2
• Component/s: Connectors / Common, Connectors / Kafka

Environment

Kafka 0.8.2 and Flink 1.3.2 on YARN mode

• Assignee: Unassigned
• Reporter: Weihua Jiang
• Created: 08/Nov/17 08:31
• Updated: 28/Feb/19 10:46
• Resolved: 13/Feb/18 09:58

Description

Our streaming job run into trouble in these days after a long time smooth running. One issue we found is https://issues.apache.org/jira/browse/FLINK-8019 and another one is this one.

After analyzing the jstack, we believe we found a DEAD LOCK in flink:
1. The thread "cache-process0 -> async-operator0 -> Sink: hbase-sink0 (8/8)" hold lock 0x00000007b6aa1788 and is waiting for lock 0x00000007b6aa1940.
2. The thread "Time Trigger for cache-process0 -> async-operator0 -> Sink: hbase-sink0 (8/8)" hold lock 0x00000007b6aa1940 and is waiting for lock 0x00000007b6aa1788.

This DEADLOCK made the job fail to proceed.

Issue Links

• is duplicated by: FLINK-8019 Flink streaming job stopped at consuming Kafka data