Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2163

Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Pending Closed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: None
    • Labels:
      None

      Description

      When leadership of an offsets partition moves, the new leader loads offsets from that partition into the offset manager cache.
      Independently, the offset manager has a periodic cleanup task for stale offsets that removes old offsets from the cache and appends tombstones for those. If the partition happens to contain much older offsets (earlier in the log) and inserts those into the cache; the cleanup task may run and see those offsets (which it deems to be stale) and proceeds to remove from the cache and append a tombstone to the end of the log. The tombstone will override the true latest offset and a subsequent offset fetch request will return no offset.
      We just need to prevent the cleanup task from running during an offset load.

        Attachments

        1. KAFKA-2163.patch
          18 kB
          Joel Koshy

          Activity

            People

            • Assignee:
              jjkoshy Joel Koshy
              Reporter:
              jjkoshy Joel Koshy
              Reviewer:
              Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: