[SAMZA-1282] Spinning up more containers than the number of tasks kills leader

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.13.1
• Component/s: container
• Assignee: Shanthoosh Venkataraman
• Reporter: Bharath Kumarasubramanian
• Created: 10/May/17 22:54
• Updated: 25/Jul/17 22:32
• Resolved: 25/Jul/17 22:32

Description

When a user tries to spin up more containers than the max partitions or tasks, the leader process gets killed.

We throw an exception in the TaskNameGrouper for the above scenario and that needs to be handled gracefully by the leader and kill the newly spun containers as opposed bailing out.

Here is the stack trace

 2017-05-10 15:13:24.526 [debounce-thread-0] ScheduleAfterDebounceTime [ERROR] OnProcessorChange threw an exception.
java.lang.IllegalArgumentException: number of containers 2 is bigger than number of tasks 1
	at org.apache.samza.container.grouper.task.GroupByContainerIds.group(GroupByContainerIds.java:68)
	at org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:258)
	at org.apache.samza.coordinator.JobModelManager.readJobModel(JobModelManager.scala)
	at org.apache.samza.zk.ZkJobCoordinator.generateNewJobModel(ZkJobCoordinator.java:212)
	at org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:125)
	at org.apache.samza.zk.ZkJobCoordinator.lambda$onProcessorChange$1(ZkJobCoordinator.java:120)
	at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$scheduleAfterDebounceTime$0(ScheduleAfterDebounceTime.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• relates to: SAMZA-1064 Standalone Samza with Zookeeper for Coordination
• links to: GitHub Pull Request #244