Description
KIP-213: https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable
Today in Kafka Streams DSL, KTable joins are only based on keys. If users want to join a KTable A by key a with another KTable B by key b but with a "foreign key" a, and assuming they are read from two topics which are partitioned on a and b respectively, they need to do the following pattern:
tableB' = tableB.groupBy(/* select on field "a" */).agg(...); // now tableB' is partitioned on "a"
tableA.join(tableB', joiner);
Even if these two tables are read from two topics which are already partitioned on a, users still need to do the pre-aggregation in order to make the two joining streams to be on the same key. This is a draw-back from programability and we should fix it.
Attachments
Issue Links
- is a parent of
-
KAFKA-9302 Allow custom partitioning in table-table joins
-
- Open
-
- is related to
-
KAFKA-9390 Non-key joining of KTable not compatible with confluent avro serdes
-
- Resolved
-
-
KAFKA-9138
Add system test covering Foreign Key joins (KIP-213)
-
- Resolved
-
- links to
