[SAMZA-2161] Move ChangelogPartitionManager and CoordinatorStream ConfigReader to MetadataStore

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 09/Apr/19 23:21
• Updated: 17/May/19 19:26
• Resolved: 17/May/19 19:26

Description

Currently the metadata of a samza job is stored into a kafka topic named coordinator stream.

In samza-yarn ApplicationMaster, the same coordinator stream is read twice as a part of the startup sequence. This duplicate read unnecessarily prolongs the startup time of the application master and makes the container allocation take longer than usual. This inadvertently incurs a substantial increase in input stream processing delay depending upon the size of the coordinator stream. 

To mitigate this problem, the two util classes in samza viz `ChangelogPartitionManager`, `Config` should be moved to use the MetadataStore abstraction. This ticket tracks the work involved in the migration. 

Issue Links

• links to: GitHub Pull Request #990