Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9177

Pause completed partitions on restore consumer

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: streams
    • Labels:
      None

      Description

      The StoreChangelogReader is responsible for tracking and restoring active tasks, but once a store has finished restoring it will continue polling for records on that partition.

      Ordinarily this doesn't make a difference as a store is not completely restored until its entire changelog has been read, so there are no more records for poll to return anyway. But if the restoring state is actually an optimized source KTable, the changelog is just the source topic and poll will keep returning records for that partition until all stores have been restored.

      Note that this isn't a correctness issue since it's just the restore consumer, but it is wasteful to be polling for records and throwing them away. We should pause completed partitions in StoreChangelogReader so we don't slow down the restore consumer in reading from the unfinished changelog topics, and avoid wasted network.

        Attachments

          Activity

            People

            • Assignee:
              guozhang Guozhang Wang
              Reporter:
              ableegoldman A. Sophie Blee-Goldman
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: