Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19537

Processed in-flight bytes metric is not accurate

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.12.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      Processed in-flight bytes as introduced in FLINK-18662 is not entirely accurate, as it's ignoring the buffer/bytes accumulated in the record deserializers. If buffer is processed here, it doesn't mean it was fully processed (so we can over estimate the amount of processed bytes). On the other hand some records/bytes might be processed without polling anything from this CheckpointedInputGate (underestimating the amount of processed bytes). All in all this should have been calculated on the StreamTaskNetworkInput level, where we have an access to the records deserializers. However the current is on average accurate and it might be just good enough (at least for the time being).

      Also this metric is currently ignoring chained source inputs to the multiple input stream task.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pnowojski Piotr Nowojski
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: