Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-255

Rewinding Streams within a StreamTask

    XMLWordPrintableJSON

    Details

    • Type: Wish
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The many benefits of Kafka include persistent storage and its resulting possibility to rewind streams to a specific offset. Samza does currently not support rewinding of streams within a StreamTask. I'd like to place this functionality as a feature request and provide two use cases to further describe the benefits of such a feature. Let's consider a general use case to aggregate values within sliding windows.

      1. Offline-Processing
      In case of offline-processing the sliding window does not correlate to the system time. In this case any node failure will result in samza restoring from a checkpointed offset that most probably does not match the beginning of the most recent sliding window. But in order to gain precise results, one could rewind to the specific offset and process the missing events of the sliding window. The same holds for any use case where the data has to be processed in small batches and these batches do not correspond to the system time.

      2. Late Arrival
      Messages might get delayed before they are stored into Kafka. In this case one could rewind the offset in order to process older messages corresponding to the same sliding window.

      I'd be happy to further discuss these cases and the proposed feature request.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jagadish1989@gmail.com Jagadish
                Reporter:
                nicolas.baer Nicolas Bär
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: