Affects Version/s: None
Fix Version/s: None
Symptom: Checkpoint and changelog kafka topics are expected to be created for a samza job with cleanup.policy set to compact. This ensures that all the data in the metadata topics are log-compacted. However, we observed that checkpoint & changelog topics for hundreds of production samza jobs were created with cleanup.policy set to delete leading to strong-possibility of message-l0ss.
- Checkpoint: The control-flow in KafkaStreamSpec to build checkpoint spec swallows the essential kafka-topic configuration rather passes empty configuration bag to kafka-broker(via kafka-rest). This issue was introduced in
- Changelog: There change-log topic configurations are incorrectly generated when the RocksDB store TTL is set to -1 by the user.
Changes: Change the topic-creation control-flow for above metadata topics to generate correct topic-configurations.
Tests: Added unit tests to validate the expected topic configuration bag.
API Changes: None
Upgrade Instructions: None
Usage Instructions: None