Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-15417

JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.8.0
    • streams
    • None

    Description

      In Kafka-streams 3.4.0 :

      According to the javadoc of the Joinwindow:

      There are three different window configuration supported:

      • before = after = time-difference
      • before = 0 and after = time-difference
      • before = time-difference and after = 0

       

      However if I use a joinWindow with before = time-difference and after = 0
      on a kstream-kstream-leftjoin the after=0 part does not seem to work.

      When using stream1.leftjoin(stream2, joinWindow) with joinWindow.after=0 and joinWindow.before=30s, any new message on stream 1 that can not be joined with any messages on stream2 should be joined with a null-record after the joinWindow.after has ended and a new message has arrived on stream1.

      It does not.

      Only if the new message arrives after the value of joinWindow.before the previous message will be joined with a null-record.

       

      Attached you can find two files with a TopologyTestDriver Unit test to reproduce.

      topology:   stream1.leftjoin( stream2, joiner, joinwindow)

      joinWindow has before=5000ms and after=0

      message1(key1) ->  stream1

      after 4000ms message2(key2) -> stream1  ->  NO null-record join was made, but the after period was expired.
      after 4900ms message2(key2) -> stream1  ->  NO null-record join was made, but the after period was expired.
      after 5000ms message2(key2) -> stream1  ->  A null-record join was made,  before period was expired.
      after 6000ms message2(key2) -> stream1  ->  A null-record join was made,  before period was expired.

       

      Attachments

        1. Afbeelding 1.png
          21 kB
          Victor van den Hoven
        2. Afbeelding 1-1.png
          64 kB
          Victor van den Hoven
        3. SimpleStreamTopology.java
          2 kB
          Victor van den Hoven
        4. SimpleStreamTopologyTest.java
          3 kB
          Victor van den Hoven

        Activity

          People

            VictorvandenHoven Victor van den Hoven
            VictorvandenHoven Victor van den Hoven
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: