Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2000

Delete consumer offsets from kafka once the topic is deleted

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.2.0, 0.11.0.0
    • Component/s: None
    • Labels:
    1. KAFKA-2000_2015-05-03_10:39:11.patch
      4 kB
      Sriharsha Chintalapani
    2. KAFKA-2000.patch
      1 kB
      Sriharsha Chintalapani

      Issue Links

        Activity

        Hide
        sriharsha Sriharsha Chintalapani added a comment -

        Created reviewboard https://reviews.apache.org/r/32650/diff/
        against branch origin/trunk

        Show
        sriharsha Sriharsha Chintalapani added a comment - Created reviewboard https://reviews.apache.org/r/32650/diff/ against branch origin/trunk
        Hide
        sriharsha Sriharsha Chintalapani added a comment -

        Updated reviewboard https://reviews.apache.org/r/32650/diff/
        against branch origin/trunk

        Show
        sriharsha Sriharsha Chintalapani added a comment - Updated reviewboard https://reviews.apache.org/r/32650/diff/ against branch origin/trunk
        Hide
        ijuma Ismael Juma added a comment -

        Is this still applicable?

        Show
        ijuma Ismael Juma added a comment - Is this still applicable?
        Hide
        ijuma Ismael Juma added a comment -

        Looks like it is, `GroupMetadataManager` has the following code:

        val expiredOffsets = offsetsCache.filter { case (groupTopicPartition, offsetAndMetadata) =>
          offsetAndMetadata.expireTimestamp < startMs
        }
        

        sriharsha chintalapani, do you want to submit a PR with this change?

        Show
        ijuma Ismael Juma added a comment - Looks like it is, `GroupMetadataManager` has the following code: val expiredOffsets = offsetsCache.filter { case (groupTopicPartition, offsetAndMetadata) => offsetAndMetadata.expireTimestamp < startMs } sriharsha chintalapani , do you want to submit a PR with this change?
        Hide
        sriharsha Sriharsha Chintalapani added a comment -
        Show
        sriharsha Sriharsha Chintalapani added a comment - Ismael Juma Yes.
        Hide
        parth.brahmbhatt Parth Brahmbhatt added a comment -

        Talked to sriharsha chintalapani , I will submit an updated patch for this one today.

        Show
        parth.brahmbhatt Parth Brahmbhatt added a comment - Talked to sriharsha chintalapani , I will submit an updated patch for this one today.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user Parth-Brahmbhatt opened a pull request:

        https://github.com/apache/kafka/pull/704

        KAFKA-2000: Delete topic should also delete consumer offsets.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2000

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/kafka/pull/704.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #704


        commit ceae0b7031d297a7db6664b435bb3cdc55228646
        Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
        Date: 2015-12-18T20:35:32Z

        KAFKA-2000: Delete topic should also delete consumer offsets.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/kafka/pull/704 KAFKA-2000 : Delete topic should also delete consumer offsets. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2000 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/704.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #704 commit ceae0b7031d297a7db6664b435bb3cdc55228646 Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com> Date: 2015-12-18T20:35:32Z KAFKA-2000 : Delete topic should also delete consumer offsets.
        Hide
        sslavic Stevo Slavic added a comment -

        I'm still on 0.8.2.x and it's hurting my system smoke tests, reusing same topics over and over again in the test, (consumer) state preserved - deleting topic, creating it, publishing message, not being able to read just published message. Now have to introduce dummy read after topic is recreated, just to have existing offset fall outside of the valid range, and get reset.

        Curious, are there any plans to backport this fix to 0.9.0.x or even 0.8.2.x?

        Show
        sslavic Stevo Slavic added a comment - I'm still on 0.8.2.x and it's hurting my system smoke tests, reusing same topics over and over again in the test, (consumer) state preserved - deleting topic, creating it, publishing message, not being able to read just published message. Now have to introduce dummy read after topic is recreated, just to have existing offset fall outside of the valid range, and get reset. Curious, are there any plans to backport this fix to 0.9.0.x or even 0.8.2.x?
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user omkreddy opened a pull request:

        https://github.com/apache/kafka/pull/1850

        KAFKA-2000: Delete topic should also delete consumer offsets.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/omkreddy/kafka KAFKA-2700-DELETE

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/kafka/pull/1850.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #1850


        commit 3bcb8232b6f184e232797519b1a829e74d0f37b1
        Author: Manikumar Reddy O <manikumar.reddy@gmail.com>
        Date: 2016-09-13T16:07:41Z

        KAFKA-2000: Delete topic should also delete consumer offsets.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/1850 KAFKA-2000 : Delete topic should also delete consumer offsets. You can merge this pull request into a Git repository by running: $ git pull https://github.com/omkreddy/kafka KAFKA-2700 -DELETE Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1850.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1850 commit 3bcb8232b6f184e232797519b1a829e74d0f37b1 Author: Manikumar Reddy O <manikumar.reddy@gmail.com> Date: 2016-09-13T16:07:41Z KAFKA-2000 : Delete topic should also delete consumer offsets.
        Hide
        hachikuji Jason Gustafson added a comment -

        It would be nice to get this fixed. The patch from Parth Brahmbhatt has review comments from Feb. 2016 which have not been addressed. Perhaps Manikumar can rebase the patch posted above and we can try to get it merged?

        Show
        hachikuji Jason Gustafson added a comment - It would be nice to get this fixed. The patch from Parth Brahmbhatt has review comments from Feb. 2016 which have not been addressed. Perhaps Manikumar can rebase the patch posted above and we can try to get it merged?
        Hide
        jeffwidman Jeff Widman added a comment - - edited

        If neither of them is interested, I'm happy to cleanup the existing patch to get it merged into 0.10.2. The test suite at my work would benefit from this. Just let me know.

        Show
        jeffwidman Jeff Widman added a comment - - edited If neither of them is interested, I'm happy to cleanup the existing patch to get it merged into 0.10.2. The test suite at my work would benefit from this. Just let me know.
        Hide
        sriharsha Sriharsha Chintalapani added a comment -

        Jeff Widman

        Manikumar working on it.

        Show
        sriharsha Sriharsha Chintalapani added a comment - Jeff Widman Manikumar working on it.
        Hide
        sriharsha Sriharsha Chintalapani added a comment -

        Issue resolved by pull request 1850
        https://github.com/apache/kafka/pull/1850

        Show
        sriharsha Sriharsha Chintalapani added a comment - Issue resolved by pull request 1850 https://github.com/apache/kafka/pull/1850
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/kafka/pull/1850

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1850

          People

          • Assignee:
            omkreddy Manikumar
            Reporter:
            sriharsha Sriharsha Chintalapani
            Reviewer:
            Jason Gustafson
          • Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development