We currently provide auto-create for changelog streams. However, when there are more than 1 containers, it is possible that Samza creates a changelog stream with insufficient partitions.
assume we have an input stream with 3 partitions and then we assign 3 containers for this job. According to the JobCoordinator, we will get:
|Container(Model)||InputStream Partition||Changelog Partition|
If Container 0 is brought up first, it calls
The maxChangeLogStreamPartition is 1. So we will auto-create a changelog stream with only 1 partitions.
Similarly, if the Container 2 is brought up first, we will get a stream with 2 partitions.