[FLINK-32127] Source busy time is inaccurate in many cases

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: kubernetes-operator-1.7.0
• Component/s: Autoscaler
• Assignee: Maximilian Michels
• Reporter: Zhanghao Chen
• Created: 18/May/23 06:16
• Updated: 15/Nov/23 12:39
• Resolved: 15/Nov/23 12:39

Description

We found that source busy time is inaccurate in many cases. The reason is that sources are usu. multi-threaded (Kafka and RocketMq for example), there is a fetcher thread fetching data from data source, and a consumer thread deserializes data with an blocking queue in between. A source is considered 

1. idle if the consumer is blocked by fetching data from the queue
2. backpressured if the consumer is blocked by writing data to downstream operators
3. busy otherwise

However, this means that if the bottleneck is on the fetcher side, the consumer will be often blocked by fetching data from the queue, the source idle time would be high, but in fact it is busy and consumes a lot of CPU. In some of our jobs, the source max busy time is only ~600 ms while it has actually reached the limit.

The bottleneck could be on the fetcher side, for example, when Kafka enables zstd compression, uncompression on the consumer side could be quite heavy compared to data deserialization on the consumer thread side.

Issue Links

• is related to: FLINK-33306 Use observed true processing rate when source metrics are incorrect