[FLINK-12373] Improve checkpointing metrics

• Type: New Feature
• Status: Reopened
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Runtime / Checkpointing
• Assignee: Unassigned
• Reporter: Gyula Fora
• Created: 30/Apr/19 09:33
• Updated: 29/Jul/21 07:38

Description

The checkpoint metrics encapsulated in the CheckpointMetrics class currently exposes 4 core metrics for each operator: bytesBuffered, alignment time, sync duration and async duration

I think it would be a great improvement to break up the tracking of the sync duration into the different components as it contains information that is critical to improve the SLA of large jobs.

I suggest we break up the sync duration into 4 subcomponents:

 1. prepareSnapshotPreBarrier
 2. Snapshot timers
 3. Snapshot operator states
 4. Sync keyed state checkpoint

Maybe the operator state part could be further broken up into keyed/non-keyed part, i dont know.

I think knowing these metrics is crucial for users to minimise the latency caused by checkpointing.

Whether we want to show all this info on the web ui is another discussion