Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: streams
    • Labels:
      None

      Description

      Today whenever users specify a queryable store name for KTable, we would always add a physical state store in the translated processor topology.

      For some scenarios, we should consider not physically materialize the KTable but only "logically" materialize it when you have some simple transformation operations or even join operations that generated new KTables, and which needs to be materialized with a state store, you can use the changelog topic of the previous KTable and applies the transformation logic upon restoration instead of creating a new changelog topic. For example:

      table1 = builder.table("topic1");
      table2 = table1.filter(..).join(table3); // table2 needs to be materialized for joining
      

      We can actually set the getter function of table2's materialized store, say state2 to be reading from topic1 and then apply the filter operator, instead of creating a new state2-changelog topic in this case.

      We can come up with a general internal impl optimizations to determine when to logically materialize, and whether we should actually allow users of DSL to "hint" whether to materialize or not (it then may need a KIP).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                guozhang Guozhang Wang
                Reporter:
                guozhang Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: