Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-348 Configure Samza jobs through a stream
  3. SAMZA-465

Use coordinator stream and eliminate CheckpointManager

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: container, kafka
    • Labels:
      None

      Description

      SAMZA-448 introduced a coordinator stream into Samza, but still kept CheckpointManagers. This means that master currently has two streams, which is totally unnecessary. We should eliminate the CheckpointManager, and use the coordinator stream instead.

      This work will involve modifying the JobCoordinator to continue reading messages from the coordinator stream even after it starts up. This is required because a container may fail and restart. When it does so, the offset that it receives from the JobCoordinator (via the HTTP JSON server) must be up to date.

        Attachments

        1. SAMZA-465.patch
          411 kB
          Naveen Somasundaram
        2. 0001-Coordinator-stream.patch
          280 kB
          Naveen Somasundaram
        3. 0001-Coordinator-stream.patch
          280 kB
          Naveen Somasundaram
        4. 0001-Coordinator-stream.patch
          290 kB
          Naveen
        5. 0001-Coordinator-stream.patch
          290 kB
          Naveen
        6. 0001-Coordinator-stream.patch
          219 kB
          Naveen
        7. 0001-Coordinator-stream.patch
          219 kB
          Naveen
        8. 0001-Coordinator-stream.patch
          291 kB
          Naveen
        9. 0001-Coordinator-stream.patch
          291 kB
          Naveen Somasundaram

          Activity

            People

            • Assignee:
              naveenatceg Naveen Somasundaram
              Reporter:
              criccomini Chris Riccomini
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: