[FLINK-33306] Use observed true processing rate when source metrics are incorrect - ASF JIRA

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: kubernetes-operator-1.7.0
Component/s: Kubernetes Operator
Labels:
- pull-request-available

Description

The aim is to address the cases when Flink incorrectly reports low busy time (high idleness) for sources that are in fact cannot keep up due to the slowness of the reader/fetchers. As the metrics cannot be generally fixed on the Flink - connector side we have to detect this and handle it when collecting the metrics.

The main symptom of this problem is overestimation of the true processing rate and not triggering scaling even if lag is building up as the autoscaler thinks it will be able to keep up.

To tackle this we differentiate two different methods of TPR measurement:

Busy-time based TPR (this is the current approach in the autoscaler) : computed from incoming records and busy time
Observed TPR : computed from incoming records and back pressure, measurable only when we assume full processing throughput (i.e during catch-up)

Current behaviour

The operator currently always uses a busy-time based TPR calculation which is very flexible and allows for scaling up / down but is susceptible to overestimation due to the broken metrics.

Suggested new behaviour

Instead of using the busy-time based TPR we detect when TPR is overestimated (busy-time too low) and switch to observed TPR.

To do this, whenever we there is lag for a source (during catchup, or lag-buildup) we measure both busy-time and observed TPR.

If the avg busy-time based TPR is off by a configured amount we switch to observed TPR for this source during metric evaluation.

Why not use observed TPR all the time?
Observed TPR can only be measured when we are catching up (during stabilization) or when cannot keep up. This makes it harder to scale down or to detect changes in source throughput over time (before lag starts to build up). Instead of using observed TPR we switch to it only when we detect a problem with the busy-time (this is a rare case overall), to hopefully get the best of both worlds.