Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33306

Use observed true processing rate when source metrics are incorrect

    XMLWordPrintableJSON

Details

    Description

      The aim is to address the cases when Flink incorrectly reports low busy time (high idleness) for sources that are in fact cannot keep up due to the slowness of the reader/fetchers. As the metrics cannot be generally fixed on the Flink - connector side we have to detect this and handle it when collecting the metrics.

      The main symptom of this problem is overestimation of the true processing rate and not triggering scaling even if lag is building up as the autoscaler thinks it will be able to keep up.

      To tackle this we differentiate two different methods of TPR measurement:

      1. Busy-time based TPR (this is the current approach in the autoscaler) : computed from incoming records and busy time
      2. Observed TPR : computed from incoming records and back pressure, measurable only when we assume full processing throughput (i.e during catch-up)

      Current behaviour

      The operator currently always uses a busy-time based TPR calculation which is very flexible and allows for scaling up / down but is susceptible to overestimation due to the broken metrics.

      Suggested new behaviour

      Instead of using the busy-time based TPR we detect when TPR is overestimated (busy-time too low) and switch to observed TPR.

      To do this, whenever we there is lag for a source (during catchup, or lag-buildup) we measure both busy-time and observed TPR.

      If the avg busy-time based TPR is off by a configured amount we switch to observed TPR for this source during metric evaluation.

      Why not use observed TPR all the time?
      Observed TPR can only be measured when we are catching up (during stabilization) or when cannot keep up. This makes it harder to scale down or to detect changes in source throughput over time (before lag starts to build up). Instead of using observed TPR we switch to it only when we detect a problem with the busy-time (this is a rare case overall), to hopefully get the best of both worlds.

      Attachments

        Issue Links

          Activity

            People

              gyfora Gyula Fora
              gyfora Gyula Fora
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: