Major Description
Among others, Streams reports the following two thread-level "process" metrics:
"process-rate": The average number of process calls per second.
"process-total": The total number of process calls across all tasks.
See the docs: https://kafka.apache.org/documentation/#kafka_streams_thread_monitoring
There's some surprising ambiguity in these definitions that has led to Streams actually reporting something different than what most people would probably expect. Specifically, it's not defined what a "process call" is.
A reasonable definition of a "process call" is processing a record or processing a task (both of which are publicly facing concepts, and both of which are the same, since tasks process records one at a time). However, we currently measure number of invocations to a private, internal `process()` method, which would actually process more than one record at a time. Thus, the current metric is under-counting the throughput, in an esoteric and confusing way.
Instead, we should simply change the rate and total metrics to measure the (rate and total) of record processing.
Attachments
Issue Links
- links to