Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-157

Offset default behavior for streams

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.7.0
    • container
    • None

    Description

      Introduce a systems.<system name>.streams.<stream name>.samza.offset.default configuration, which specifies what to do when no checkpoint exists for an input topic. This is a similar to setting to Kafka's auto.offset.reset setting. Developers will be able to specify "oldest", "latest", or "fail".

      we should also add the ability to override offsets for specific stream partitions. Something like:

      systems.<system name>.streams.<stream name>.samza.force.offsets=0:123,1:123,2:123
      

      The format I'm proposing is:

      <partition string>:<force offset>,...
      

      This is obviously dependent on offsets not having ':' or ',' in them, which I think is a safe assumption.

      This setting would force the system consumer to be registered with the specified offset for the given SSP (ignoring both the checkpoint, if it exists, and the samza.reset.offset setting).

      Attachments

        1. SAMZA-157.0.patch
          66 kB
          Chris Riccomini
        2. SAMZA-157.1.patch
          41 kB
          Chris Riccomini
        3. SAMZA-157.3.patch
          55 kB
          Chris Riccomini
        4. SAMZA-157.4.patch
          54 kB
          Chris Riccomini
        5. SAMZA-157.5.patch
          60 kB
          Chris Riccomini

        Issue Links

          Activity

            People

              criccomini Chris Riccomini
              criccomini Chris Riccomini
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: