Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20374 Wrong result when shuffling changelog stream on non-primary-key columns
  3. FLINK-23993

Describe eventually-consistency of materializing upserts

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.14.0
    • None
    • Documentation, Table SQL / Ecosystem
    • None

    Description

      FLINK-20374 added an upsert materialization operator which fixes the order of shuffled streams. The results of this operator are actually eventually consistent (it collects the latest value it has seen and redacts older versions when these are not valid anymore). You could see a result stream like this, based on the order the materialization receives events:

      +I10, -I10, +I5, -I5, +I10, -I10, +I3, -I3, +I10

      Each time, the value stored in Kafka would change until the "final" result is in.

       

      It may be acceptable for upsert sinks, but should be documented (or changed/fixed) nonetheless.
       

      Attachments

        Activity

          People

            Unassigned Unassigned
            nkruber Nico Kruber
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: