[FLINK-3127] Measure backpressure in Flink jobs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.0, 0.10.1
• Fix Version/s: 1.0.0
• Component/s: Runtime / Coordination, Runtime / Web Frontend
• Assignee: Unassigned
• Reporter: Maximilian Michels
• Created: 07/Dec/15 10:10
• Updated: 28/Feb/19 10:52
• Resolved: 03/Feb/16 09:09

Description

Multiple users have asked for a simple way to display the backpressure in a Flink job. Backpressure could be determined by monitoring throughput and allocated network buffers at the task managers.

Subsequently, a reliable measure of backpressue could be used to adjust the read behavior of sources, e.g. adjust the offset of the Kafka consumer.

Also, the web frontend should display the backpressure measure to allow users to find bottlenecks in there jobs.

Issue Links

• is duplicated by: FLINK-3310 Add back pressure statistics to web frontend