Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
In some cases it would be useful for systems outside Kafka to have the ability to know when Kafka deletes records (tombstoning or retention).
In general the use-case is where there is a desire to link the lifecycle of a record in a third party system (database or filesystem etc) to the lifecycle of the record in Kafka.
A concrete use-case: a system using Kafka to distribute video clips + metadata. The binary content is too big to store in Kafka so the publishing application caches the content in cloud storage and publishes a record containing a S3 url to the video clip. The desire is to have a mechanism to remove the clip from cloud storage at the same time the record is expunged from Kafka by retention or tombstoning. Currently there is no practical way to achieve this.
Desired solution
A pluggable broker-side mechanism that is informed as records are being compacted away or deleted. The API would expose the topic from which the record is being deleted, the record key, record headers, timestamp and (possibly) record value.