Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-971

JobCoordinator/JobModelManager does not need to fetch offset for all stream partitions

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.1
    • Component/s: None
    • Labels:

      Description

      JobCoordinator / JobModelManager does not need to fetch offset for all stream partitions. It only needs the partition count for each stream in order distribute them among tasks.

      The impact of fetching offsets is that when many topic partitions are being consumed, it takes longer for the Samza job to boot-up. If the yarn-am-liveness timeout is set to be lower than the time for the AM to boot up, then the RM kills the application. Such a job may never be able to start-up.

      The main problem here is the generic interface in SystemAdmin - getSystemStreamMetadata for fetching partition count AND offset information. If we have separate interfaces for fetching each of these information, it will provide more granular control on fetching only required information. A similar approach was used in SAMZA-882 to detect the partition count changes in the input streams.

        Attachments

        1. SAMZA-971-0.patch
          0.7 kB
          Navina Ramesh
        2. SAMZA-971-1.patch
          1 kB
          Navina Ramesh
        3. SAMZA-971-2.patch
          2 kB
          Navina Ramesh

          Activity

            People

            • Assignee:
              navina Navina Ramesh
              Reporter:
              navina Navina Ramesh
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: