[SAMZA-586] LoggedStore can send compressed messages to log-compacted Kafka topic

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: kafka
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 05/Mar/15 23:10
• Updated: 19/Mar/15 16:26
• Resolved: 19/Mar/15 16:26

Description

I think we might have an issue with changelogs that use Kafka as its backing store. The Kafka topics are log-compacted. If the system is configured to use compression (systems.<sysname>.producer.compression.type=gzip), I believe that the LoggedStore will send compressed messages to the log-compacted topic. The LoggedStore shares the normal producer based on system name. Due to KAFKA-1374, this will cause the log compaction thread to die, or at the very least, not compress the topic.

Issue Links

• relates to: KAFKA-1374 LogCleaner (compaction) does not support compressed topics
• relates to: SAMZA-398 Enable compression in Kafka checkpoint and changelog topics