Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7658

Add KStream#toTable to the Streams DSL

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      KIP-523: https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL

       

      We'd like to add a new API to the KStream object of the Streams DSL:

      KTable KStream.toTable()
      
      KTable KStream.toTable(Materialized)
      

      The function re-interpret the event stream KStream as a changelog stream KTable. Note that this should NOT be treated as a syntax-sugar as a dummy KStream.reduce() function which always take the new value, as it has the following difference:

      1) an aggregation operator of KStream is for aggregating a event stream into an evolving table, which will drop null-values from the input event stream; whereas a toTable function will completely change the semantics of the input stream from event stream to changelog stream, and null-values will still be serialized, and if the resulted bytes are also null they will be interpreted as "deletes" to the materialized KTable (i.e. tombstones in the changelog stream).

      2) the aggregation result KTable will always be materialized, whereas toTable resulted KTable may only be materialized if the overloaded function with Materialized is used (and if optimization is turned on it may still be only logically materialized if the queryable name is not set).

      Therefore, for users who want to take a event stream into a changelog stream (no matter why they cannot read from the source topic as a changelog stream KTable at the beginning), they should be using this new API instead of the dummy reduction function.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ash26389 Aishwarya Pradeep Kumar
                Reporter:
                guozhang Guozhang Wang
              • Votes:
                3 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: