Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.2.0
    • Fix Version/s: 1.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      In KIP-99 we have added support for GlobalKTables, however we don't currently support KTable/GlobalKTable joins as they require materializing a state store for the join.

        Issue Links

          Activity

          Hide
          bbejeck Bill Bejeck added a comment -

          Damian Guy Is it ok for me to pick this one up? I wasn't sure if planned on doing this one as part of your KIP-99 work.

          Show
          bbejeck Bill Bejeck added a comment - Damian Guy Is it ok for me to pick this one up? I wasn't sure if planned on doing this one as part of your KIP-99 work.
          Hide
          damianguy Damian Guy added a comment -

          Bill Bejeck thanks for getting in touch. We'd like to wait for the outcome of the KIP-114 discussion before looking at this. As noted above, this requires that we materialize the state store and we want to make sure that whatever is done is consistent with KTable. So, it is probably not the best JIRA to pick up at this point in time.

          Show
          damianguy Damian Guy added a comment - Bill Bejeck thanks for getting in touch. We'd like to wait for the outcome of the KIP-114 discussion before looking at this. As noted above, this requires that we materialize the state store and we want to make sure that whatever is done is consistent with KTable. So, it is probably not the best JIRA to pick up at this point in time.
          Hide
          dminkovsky Dmitry Minkovsky added a comment - - edited

          on KAFKA-4880 (a dupe of this ticket), @Michael Noll writes:

          > FWIW, I wonder how much interest there actually is for this functionality. Users have been requesting stream-globalTable joins, but personally I have yet to run into a person that wants table-globalTable joins. Just saying.

          I hadn't noticed that GlobalKTables were severely limited and assumed they had all the functionality of regular tables. I had been really excited to use them for "hydration"/denormalization. I have two models which will be present in my application in different quantities. Instances of one model will be very few in quantity compared to instances of the second model. I want to "hydrate" instances of the high-quantity model with instances of the low-quantity model, which can be available in its entirety on all streams app instances. KTable/KTable joins are great for denormalization because you can drive the update from both sides of the relationship. KTable/GlobalKTable joins would facilitates such updates while removing the need to repartition the left table.

          Show
          dminkovsky Dmitry Minkovsky added a comment - - edited on KAFKA-4880 (a dupe of this ticket), @Michael Noll writes: > FWIW, I wonder how much interest there actually is for this functionality. Users have been requesting stream-globalTable joins, but personally I have yet to run into a person that wants table-globalTable joins. Just saying. I hadn't noticed that GlobalKTables were severely limited and assumed they had all the functionality of regular tables. I had been really excited to use them for "hydration"/denormalization. I have two models which will be present in my application in different quantities. Instances of one model will be very few in quantity compared to instances of the second model. I want to "hydrate" instances of the high-quantity model with instances of the low-quantity model, which can be available in its entirety on all streams app instances. KTable/KTable joins are great for denormalization because you can drive the update from both sides of the relationship. KTable/GlobalKTable joins would facilitates such updates while removing the need to repartition the left table.
          Hide
          guozhang Guozhang Wang added a comment -

          Dmitry Minkovsky Thanks for sharing your use cases. We are actively working on the table / global table join semantics now, stay tuned.

          Show
          guozhang Guozhang Wang added a comment - Dmitry Minkovsky Thanks for sharing your use cases. We are actively working on the table / global table join semantics now, stay tuned.
          Hide
          frankbass33 frank t added a comment -

          Guozhang Wang would be possible to release only a patch for for this ?
          or a work-around ? i am really blocked without this:.

          <K1, V1, R> KTable<K, R> join(final GlobalKTable<K1, V1> globalTable,
          final KeyValueMapper<K, V, K1> keyMapper,
          final ValueJoiner<V, V1, R> joiner);

          Show
          frankbass33 frank t added a comment - Guozhang Wang would be possible to release only a patch for for this ? or a work-around ? i am really blocked without this:. <K1, V1, R> KTable<K, R> join(final GlobalKTable<K1, V1> globalTable, final KeyValueMapper<K, V, K1> keyMapper, final ValueJoiner<V, V1, R> joiner);
          Hide
          vikasamar Vikas Tikoo added a comment -

          Is there a timeline for this feature?

          Show
          vikasamar Vikas Tikoo added a comment - Is there a timeline for this feature?

            People

            • Assignee:
              Unassigned
              Reporter:
              damianguy Damian Guy
            • Votes:
              6 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development