Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18808

Task-level numRecordsOut metric may be underestimated

    XMLWordPrintableJSON

    Details

      Description

      At present, we only register task-level numRecordsOut metric by reusing operator output record counter at the end of OperatorChain.

      if (config.isChainEnd()) {
         operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
      }
      

      If we only send data out through the last operator of OperatorChain, there is no problem with this statistics. But consider the following scenario:

      In this JobGraph, we not only send data in the last operator, but also send data in the middle operator of OperatorChain (the map operator just returns the original value directly). Below is one of our test topology, we can see that the statistics actually only have half of the total data received by the downstream.

      I think the data sent out by the intermediate operator should also be counted into the numRecordsOut of the Task. But currently we are not reusing operators output record counters in the intermediate operators, which leads to our task-level numRecordsOut metric is underestimated (although this has no effect on the actual operation of the job, it may affect our monitoring).

      A simple idea of ​​mine is to modify the condition of reusing operators output record counter:

      if (!config.getNonChainedOutputs(getUserCodeClassloader()).isEmpty()) {
         operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
      }

      In addition, I have another question: If a record is broadcast to all downstream, should the numRecordsOut counter increase by one or the downstream number? It seems that currently we are adding one to calculate the numRecordsOut metric.

        Attachments

        1. image-2020-08-04-11-28-13-800.png
          400 kB
          ming li
        2. image-2020-08-04-11-32-20-678.png
          147 kB
          ming li
        3. image-2020-08-13-18-36-13-282.png
          80 kB
          ming li

          Issue Links

            Activity

              People

              • Assignee:
                Ming Li ming li
                Reporter:
                Ming Li ming li
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: