Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
0.9.0.1, 0.10.0.0, 0.10.0.1
-
None
-
None
Description
Certain use cases, such as a streaming distinct view on data, could make use of topic compaction to retrieve a snapshot of unique messages from a compacted topic. Currently, the point to which compaction has progressed is not exposed for such use cases, though it appears to be readily available to the brokers. Without that information being exposed to consumers for such use cases, techniques such as pigging https://en.wikipedia.org/wiki/Pigging need to be implemented on an ad-hoc basis to discover the approximate offset to which compaction has progressed.
Simple API for retrieving the position of compaction for a topic partition would be useful both operationally and for use cases which need the general uniqueness guarantees offered by topic compaction.