Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-971

JobCoordinator/JobModelManager does not need to fetch offset for all stream partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.1
    • None

    Description

      JobCoordinator / JobModelManager does not need to fetch offset for all stream partitions. It only needs the partition count for each stream in order distribute them among tasks.

      The impact of fetching offsets is that when many topic partitions are being consumed, it takes longer for the Samza job to boot-up. If the yarn-am-liveness timeout is set to be lower than the time for the AM to boot up, then the RM kills the application. Such a job may never be able to start-up.

      The main problem here is the generic interface in SystemAdmin - getSystemStreamMetadata for fetching partition count AND offset information. If we have separate interfaces for fetching each of these information, it will provide more granular control on fetching only required information. A similar approach was used in SAMZA-882 to detect the partition count changes in the input streams.

      Attachments

        1. SAMZA-971-0.patch
          0.7 kB
          Navina Ramesh
        2. SAMZA-971-1.patch
          1 kB
          Navina Ramesh
        3. SAMZA-971-2.patch
          2 kB
          Navina Ramesh

        Activity

          People

            navina Navina Ramesh
            navina Navina Ramesh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: