Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4325

Improve processing of late records for window operations

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      Windows are kept until their retention time passed. If a late arriving record is processed that is older than any window kept, a new window is created containing this single late arriving record, the aggregation is computed and the window is immediately discarded afterward (as it is older than retention time).

      This behavior might case problems for downstream application as the original window aggregate might we overwritten with the late single-record- aggregate value. Thus, we should rather not process the late arriving record for this case.

      However, data loss might not be acceptable for all use cases. In order to enable the use to not lose any data, window operators should allow to register a handler function that is called instead of just dropping the late arriving record.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mjsax Matthias J. Sax
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: