Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-180

Support one-time offset reset for a Samza job

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: container
    • Labels:
      None

      Description

      Samza currently has a systems.%s.streams.%s.samza.reset.offset configuration. When set to "true", this configuration tells each SamzaContainer to disregard the checkpointed offsets for a stream when starting up. The problem with this configuration is that the checkpoints are disregarded every time the SamzaContainer starts up, not just the first time. If a host that a SamzaContainer is running on fails, and YARN (or some other mechanism) restarts the SamzaContainer, the container will not pick up where it left off, but will instead disregard the checkpointed offsets, and start over again, as before.

      There are some use-cases where developers wish to have a one-time reset of the checkpointed offsets. That is, they want to reset the offsets exactly once, but then have failures not trigger another reset. This is typically useful in bootstrapping cases (related to SAMZA-179), where a developer wishes to reset its task back to offset 0, and process all messages up to the head of a stream, then shut down. Right now, the developer can set reset.offset=true, and auto.offset.reset=smallest (if reprocessing a Kafka topic), but if the container ever restarts, processing will begin again from offset 0. This is not ideal.

        Attachments

        1. SAMZA-180.5.patch
          25 kB
          Martin Kleppmann
        2. SAMZA-180.1.patch
          13 kB
          Martin Kleppmann

          Issue Links

            Activity

              People

              • Assignee:
                martinkl Martin Kleppmann
                Reporter:
                criccomini Chris Riccomini
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: