[FLINK-21736] Introduce latency tracking state

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Release Note: State access latency metrics are introduced to track all kinds of keyed state access to help debug state performance. This feature is not enabled by default and could be turned on once setting `state.backend.latency-track.keyed-state-enabled` as true.
• Assignee: Yun Tang
• Reporter: Yun Tang
• Created: 11/Mar/21 16:16
• Updated: 28/Aug/21 11:12
• Resolved: 01/Apr/21 06:17

Description

This ticket is extended from FLINK-16444 which wants to track latency of RocksDB options. However previous ticket has two problems:

1. Cannot apply to heap based state backend and other user customized state backend.
2. User need background knowledge to know what RocksDB operation corresponding to what state operation, e.g RocksDB adopts merge to act as listState#add.

Thus, I prefer to introduce latency tracking state wrapper to detect the access latency. And to not impact the original state access, I would still use sample interval as previous FLINK-16444 did.

Issue Links

• is related to: FLINK-22528 Document latency tracking metrics for state accesses
• relates to: FLINK-16444 Count the read/write/seek/next latency of RocksDB as metrics
• links to: GitHub Pull Request #15119