Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10199

Separate state restoration into separate threads

    XMLWordPrintableJSON

Details

    Description

      As part of the restoration optimization effort, we would like to move the restoration process to separate threads such that:

      1. Stream threads would not be restricted by the main consumer `poll` frequency to keep as part of the group.

      2. We can allow larger batches of data to be written into the restoration.

      Besides this, we'd also like to fix the known issues that for piggy-backed source topics as changelog topics, the serde exception / extra processing logic would be skipped.

      We would also cleanup the global update tasks as part of this effort to consolidate to the separate restoration threads, and would also gear them up with corresponding monitoring metrics (KIPs in progress).

      Attachments

        Issue Links

          Activity

            People

              cadonna Bruno Cadonna
              guozhang Guozhang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: