Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-41

Support static partition assignment in LocalJobFactory

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.10.1
    • Component/s: container
    • Labels:

      Description

      LocalJobFactory currently creates a single container (either in ProcessJob or ThreadJob) and assigns all partitions to it using:

      val partitions = Util.getMaxInputStreamPartitions(config)
      

      This works in the case where you only wish to run a single container that processes all messages. There are situations where one container is not enough, though. If you aren't using YARN, we don't provide an easy way to run multiple containers that split partitions between them. This support would be useful for running containers in EC2, for example, where you'd wish to run two EC2 instances (for example) that host Samza containers that share partitions for a single job.

      Some potential solutions:

      1. Let developers statically assign partitions in config file.
      2. Let developers define a container ID and container count, and let LocalJobFactory/ProcessJob/ThreadJob figure out which partitions the container should own. For example, a container with id 0 and container count 2 would own partitions 0, 2, 4, 6, 8, etc.
      3. Write a different JobFactory for this case (e.g. EC2JobFactory)

        Attachments

        1. samza-41-design-doc.md
          2 kB
          Monal Daxini
        2. samza-41-design-doc.pdf
          33 kB
          Monal Daxini
        3. samza41-range.patch
          23 kB
          Monal Daxini
        4. samza41-regex.patch
          21 kB
          Monal Daxini
        5. SAMZA-41.0-updated.patch
          68 kB
          Jose
        6. rb48808.patch
          32 kB
          Yi Pan

        Issue Links

          Activity

            People

            • Assignee:
              josebarrueta Jose
              Reporter:
              criccomini Chris Riccomini

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment