[FLINK-14712] Improve back-pressure reporting mechanism

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Runtime / Metrics, Runtime / Network, Runtime / REST
• Assignee: Piotr Nowojski
• Reporter: lining
• Created: 12/Nov/19 06:30
• Updated: 13/Apr/21 08:24
• Resolved: 12/Jan/21 20:01

Description

(1) The current monitor is heavy-weight. 

•   Backpressure monitoring works by repeatedly taking stack trace samples of your running tasks.

(2) It is difficult to find out which vertex is the source  of  backpressure.

• User need to know current and upstream's network metric to judge current whether is the source of backpressure. Now user has to record relevant information.

Proposed Changes

1. expose the new mechanism implemented in FLINK-14472 as a "is back-pressured" metric.

2. show the vertex that produces the backpressure source for the job.

3. expose network metric in IOMetricsInfo:

• SubTask
  •  pool usage: outPoolUsage, inputExclusiveBuffersUsage, inputFloatingBuffersUsage.
    • If the subtask is not back pressured, but it is causing backpressure (full input, empty output)
    • By comparing exclusive/floating buffers usage, whether all channels are back-pressure or only some of them
  • back-pressured for show whether it is back pressured.
• Vertex
  • pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg
  • back-pressured for show whether it is back pressured(merge all iths subtasks)

Issue Links

• is related to: FLINK-14472 Implement back-pressure monitor with non-blocking outputs
• relates to: FLINK-22253 Update backpressure monitoring documentation
• relates to: FLINK-20724 Create a http handler for aggregating metrics from whole job

Sub-Tasks

• Expose the new mechanism implemented in FLINK-14472 as a "is back-pressured" metric: Closed
• Expose network metric for job vertex in rest api: Closed
• Expose network metric for sub task in rest api: Closed
• Show the vertex that produces the backpressure source in the job: Closed
• Better BackPressure Detection in WebUI: Closed
• Create backPressuredTimeMsPerSecond metric: Closed
• Create busyTimeMsPerSecond metrics: Closed
• Enrich back pressure stats per subtask in the WebUI: Closed
• Pause the idle/back pressure timers during processing mailbox actions: Closed