[SAMZA-1600] Changelog topics for batch jobs should not use combination of delete+compact cleanup policy in Kafka

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Assignee: Yi Pan
• Reporter: Yi Pan
• Created: 05/Mar/18 23:43
• Updated: 06/Mar/18 03:41
• Resolved: 06/Mar/18 03:41

Description

We have encountered a bug in Kafka 0.11 that when a topic is configured to use a combination of cleanup policy delete+compact, the broker enters a deadlock situation.

Currently, we set the delete+compact policy for changelog topics in batch jobs to reduce the storage requirement and the additional GC effort required. However, before the above mentioned Kafka bug is fixed, we will disable this feature for now.

Issue Links

• links to: GitHub Pull Request #435