Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8558

KIP-479 - Add StreamJoined Overload to KStream#Join

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 2.4.0
    • streams

    Description

      To prevent a topology incompatibility with the release of 2.4 and the naming of Join operations we'll add an overloaded KStream#join method accepting a Materialized parameter. This will allow users to explicitly name state stores created by Kafka Streams in the join operation.

      The overloads will apply to all flavors of KStream#join (inner, left, and right). 

      KIP-479: https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join

      Attachments

        Issue Links

          Activity

            apurva Apurva Mehta added a comment -

            Bill, where are we with fixing this? Also, if I understand correctly, as trunk stands today, if you upgrade from versions < 2.3 to trunk, and if you name your join node, you will not reuse the join state store since the name will have changed?

            apurva Apurva Mehta added a comment - Bill, where are we with fixing this? Also, if I understand correctly, as trunk stands today, if you upgrade from versions < 2.3 to trunk, and if you name your join node, you will not reuse the join state store since the name will have changed?
            apurva Apurva Mehta added a comment -

            Also, this should be a blocker for 2.4.0.

            apurva Apurva Mehta added a comment - Also, this should be a blocker for 2.4.0.
            bbejeck Bill Bejeck added a comment -

            apurva

            > Bill, where are we with fixing this?

            I've started a PR and plan to push it 8/13 - 8/14 timeframe

            >Also, if I understand correctly, as trunk stands today, if you upgrade from versions < 2.3 to trunk, and if you >name your join node, you will not reuse the join state store since the name will have changed?

            That's partially correct.  In version 2.3 - 2.1, users can name the repartition topic via `Joined.name`.  Right now in trunk if users have named the repartition topic, the same base name is used for the join operator and state store (hence changelog topic). With this Jira we'll only re-use the base name for the repartition topic and join operator and naming of the statestore (and changelog topic) is done via the Materialized object. 

             
             

            bbejeck Bill Bejeck added a comment - apurva > Bill, where are we with fixing this? I've started a PR and plan to push it 8/13 - 8/14 timeframe >Also, if I understand correctly, as trunk stands today, if you upgrade from versions < 2.3 to trunk, and if you >name your join node, you will not reuse the join state store since the name will have changed? That's partially correct.  In version 2.3 - 2.1, users can name the repartition topic via `Joined.name`.  Right now in trunk if users have named the repartition topic, the same base name is used for the join operator and state store (hence changelog topic). With this Jira we'll only re-use the base name for the repartition topic and join operator and naming of the statestore (and changelog topic) is done via the Materialized object.     
            githubbot ASF GitHub Bot added a comment -

            bbejeck commented on pull request #7285: KAFKA-8558: Add materialized to join
            URL: https://github.com/apache/kafka/pull/7285

            This PR adds `Materialized` to a `Streams` - `Streams` join, allowing to separate the naming of the join state stores from the name of the join processor. See [KIP-479 for more details](https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+Materialized+to+Join).

            For testing some of the existing tests were updated and additional tests added verifying the expected behavior.

                1. Committer Checklist (excluded from commit message)
            • [ ] Verify design and implementation
            • [ ] Verify test coverage and CI build status
            • [ ] Verify documentation (including upgrade notes)

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - bbejeck commented on pull request #7285: KAFKA-8558 : Add materialized to join URL: https://github.com/apache/kafka/pull/7285 This PR adds `Materialized` to a `Streams` - `Streams` join, allowing to separate the naming of the join state stores from the name of the join processor. See [KIP-479 for more details] ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+Materialized+to+Join ). For testing some of the existing tests were updated and additional tests added verifying the expected behavior. Committer Checklist (excluded from commit message) [ ] Verify design and implementation [ ] Verify test coverage and CI build status [ ] Verify documentation (including upgrade notes) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
            githubbot ASF GitHub Bot added a comment -

            mjsax commented on pull request #7285: KAFKA-8558: Add StreamJoined config object to join
            URL: https://github.com/apache/kafka/pull/7285

            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

            githubbot ASF GitHub Bot added a comment - mjsax commented on pull request #7285: KAFKA-8558 : Add StreamJoined config object to join URL: https://github.com/apache/kafka/pull/7285 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org

            People

              bbejeck Bill Bejeck
              bbejeck Bill Bejeck
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: