Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6034 Streams DSL to Processor Topology Translation Improvements
  3. KAFKA-6036

Enable logical materialization to physical materialization

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: streams
    • Labels:


      Today whenever users specify a queryable store name for KTable, we would always add a physical state store in the translated processor topology.

      For some scenarios, we should consider not physically materialize the KTable but only "logically" materialize it when you have some simple transformation operations or even join operations that generated new KTables, and which needs to be materialized with a state store, you can use the changelog topic of the previous KTable and applies the transformation logic upon restoration instead of creating a new changelog topic. For example:

      table1 = builder.table("topic1");
      table2 = table1.filter(..).join(table3); // table2 needs to be materialized for joining

      We can actually set the getter function of table2's materialized store, say state2 to be reading from topic1 and then apply the filter operator, instead of creating a new state2-changelog topic in this case.

      We can come up with a general internal impl optimizations to determine when to logically materialize, and whether we should actually allow users of DSL to "hint" whether to materialize or not (it then may need a KIP).


        Issue Links



            • Assignee:
              guozhang Guozhang Wang
              guozhang Guozhang Wang


              • Created:

                Issue deployment