Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3705

Support non-key joining in KTable

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.4.0
    • streams

    Description

      KIP-213: https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable

      Today in Kafka Streams DSL, KTable joins are only based on keys. If users want to join a KTable A by key a with another KTable B by key b but with a "foreign key" a, and assuming they are read from two topics which are partitioned on a and b respectively, they need to do the following pattern:

      tableB' = tableB.groupBy(/* select on field "a" */).agg(...); // now tableB' is partitioned on "a"
      
      tableA.join(tableB', joiner);
      

      Even if these two tables are read from two topics which are already partitioned on a, users still need to do the pre-aggregation in order to make the two joining streams to be on the same key. This is a draw-back from programability and we should fix it.

      Attachments

        Issue Links

          Activity

            People

              abellemare Adam Bellemare
              guozhang Guozhang Wang
              Votes:
              17 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: