Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2163

Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Pending Closed
    • None
    • 0.9.0.0
    • None
    • None

    Description

      When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache.
      Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset.
      We just need to prevent the cleanup task from running during an offset load.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jjkoshy Joel Jacob Koshy
            jjkoshy Joel Jacob Koshy
            Jun Rao Jun Rao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment