[SAMZA-2434] Fix the coordinator steam creation workflow

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0, 1.1, 1.2, 1.3
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 15/Jan/20 00:36
• Updated: 15/Jan/20 01:48

Description

Symptom: Coordinator kafka topic of a samza job currently gets created with cleanup.policy set to 'delete' instead of 'compact' for certain cases. This potentially leads to losing all the metadata(JobModel/Config) stored in coordinator stream for some samza jobs. This behavior was observed for hundreds of samza jobs within LI and would affect any job using recent samza-1.2 or samza-1.3 release in OSS.

Cause: The control-flow in KafkaSystemAdmin.createStream to build the coordinator stream spec swallows the essential kafka-topic configurations and passes along the empty configuration bag to the kafka-broker when creating the coordinator stream. This issue was introduced in SAMZA-1868.

Changes: Fix the topic-creation control-flow for the coordinator topics and generate the correct topic-configurations.

Tests: Added unit tests to validate that the expected topic configuration bag was generated for the coordinator stream topics.

API Changes: None

Upgrade Instructions: None

Usage Instructions: None
 

Issue Links

• links to: GitHub Pull Request #1252