Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-348 Configure Samza jobs through a stream
  3. SAMZA-465

Use coordinator stream and eliminate CheckpointManager

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.10.0
    • container, kafka
    • None

    Description

      SAMZA-448 introduced a coordinator stream into Samza, but still kept CheckpointManagers. This means that master currently has two streams, which is totally unnecessary. We should eliminate the CheckpointManager, and use the coordinator stream instead.

      This work will involve modifying the JobCoordinator to continue reading messages from the coordinator stream even after it starts up. This is required because a container may fail and restart. When it does so, the offset that it receives from the JobCoordinator (via the HTTP JSON server) must be up to date.

      Attachments

        1. 0001-Coordinator-stream.patch
          291 kB
          Naveen Somasundaram
        2. 0001-Coordinator-stream.patch
          291 kB
          Naveen
        3. 0001-Coordinator-stream.patch
          219 kB
          Naveen
        4. 0001-Coordinator-stream.patch
          219 kB
          Naveen
        5. 0001-Coordinator-stream.patch
          290 kB
          Naveen
        6. 0001-Coordinator-stream.patch
          290 kB
          Naveen
        7. 0001-Coordinator-stream.patch
          280 kB
          Naveen Somasundaram
        8. 0001-Coordinator-stream.patch
          280 kB
          Naveen Somasundaram
        9. SAMZA-465.patch
          411 kB
          Naveen Somasundaram

        Activity

          People

            naveenatceg Naveen Somasundaram
            criccomini Chris Riccomini
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: