[SAMZA-1989] SystemStreamGrouper interface change for SEP-5

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 12/Nov/18 07:52
• Updated: 15/Apr/19 22:48
• Resolved: 19/Mar/19 17:15

Description

Samza users may need to increase the partition count of the input streams of their stateful samza jobs. For example, Kafka needs to limit the maximum size of each partition to scale up its performance. Thus the number of partitions of a Kafka topic needs to be expanded to reduce the partition size if the average byte-in-rate or retention time of the Kafka topic has doubled.

In order to perform a join between streams, stateful jobs generally route the partitions from the different input streams to same task of a container. However, when a input stream repartitioning happens, key space of a partition gets redistributed. This will make the stateful jobs to produce erroneous results.

So if the partition count of input stream is increased then the users have to manually purge the changelog topics, local RocksDb state of their stateful jobs. This results in an increased operational complexity and data loss.

Issue Links

• is a parent of: SAMZA-1293 Enable partition expansion of input streams
• links to: GitHub Pull Request #803