Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1282

Spinning up more containers than the number of tasks kills leader

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.1
    • Component/s: container
    • Labels:
      None

      Description

      When a user tries to spin up more containers than the max partitions or tasks, the leader process gets killed.

      We throw an exception in the TaskNameGrouper for the above scenario and that needs to be handled gracefully by the leader and kill the newly spun containers as opposed bailing out.

      Here is the stack trace

       2017-05-10 15:13:24.526 [debounce-thread-0] ScheduleAfterDebounceTime [ERROR] OnProcessorChange threw an exception.
      java.lang.IllegalArgumentException: number of containers 2 is bigger than number of tasks 1
      	at org.apache.samza.container.grouper.task.GroupByContainerIds.group(GroupByContainerIds.java:68)
      	at org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:258)
      	at org.apache.samza.coordinator.JobModelManager.readJobModel(JobModelManager.scala)
      	at org.apache.samza.zk.ZkJobCoordinator.generateNewJobModel(ZkJobCoordinator.java:212)
      	at org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:125)
      	at org.apache.samza.zk.ZkJobCoordinator.lambda$onProcessorChange$1(ZkJobCoordinator.java:120)
      	at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$scheduleAfterDebounceTime$0(ScheduleAfterDebounceTime.java:89)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spvenkat Shanthoosh Venkataraman
                Reporter:
                bharathkk Bharath Kumarasubramanian
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: