Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Samza relies on a daemon thread named StreamPartitionCountMonitor which runs in the JobCoordinator to determine any change in the partition count of input streams.
Here's the control flow of StreamPartitionCountMonitor:
- JobCoordinator initializes the StreamPartitionCountMonitor thread with the partition count of the input streams.
- StreamPartitionCountMonitor periodically fetches the partition count of the input streams and compares it against the initialized partition count.
- In case of stateful jobs, if there's a difference in partition count, then it stops the JobCoordinator.
However for stateful jobs, the above solution might not work in the following scenarios:
A. In between the scheduling interval of the StreamPartitionCountMonitor, let's say that there's a change in partition count of input stream. Before the StreamPartitionCountMonitor's successive scheduled run, let's say the JobCoordinator process is killed(due to nodemanager failure). Yarn will reschedule the Jobcoordinator process to run in a different nodemanager machine. When the StreamPartitionCountMonitor is run in a different application attempt of Jobcoordinator, then it would have missed the signal of input partition count change.
B. Let's say that when the input stream partition count changes, the user stops the samza job. The daemon thread spawned in the subsequent run of the samza job would not have the adequate state to determine that the partition count of the input streams has changed.
The general pattern here is that if there's a change in partition count of input streams and JobCoordinator process is killed before the scheduled to run then we would fail to detect the partition count change.