[SAMZA-882] Detect partition count changes in input streams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.1
• Assignee: Navina Ramesh
• Reporter: Navina Ramesh
• Created: 29/Feb/16 20:10
• Updated: 25/Mar/16 02:37
• Resolved: 25/Mar/16 02:37

Description

This is a known issue where any change in the partition count in the upstream affects the Samza job and it needs to be restarted. In such scenarios, we experience data loss or incorrect processing because the application logic depends on the partitioning strategy. It is worsened by the fact that we don't even have a good mechanism to detect such a change.

As a first-step towards detection, I propose that we modify the stream metadata cache maintained in Samza such that when there a change in partition count, we increment a gauge metric. This way we can at least attach a hook to monitor when this happens and take necessary actions.

However, in the long-term, we need to come up with a better strategy for handling this.

Issue Links

• is part of: SAMZA-917 Dynamic rebalancing upon upstream changes