Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246 Update threading model for Consumer
  3. KAFKA-14274

Introduce FetchRequestManager to integrate fetch into new consumer threading refactor

    XMLWordPrintableJSON

Details

    Description

      The Fetcher class is used internally by the KafkaConsumer to fetch records from the brokers. There is ongoing work to create a new consumer implementation with a significantly refactored threading model. The threading refactor work requires a similarly refactored Fetcher.

      This task is to introduce a new class named FetchRequestManager that will be responsible for:

      1. Formatting fetch requests to the background thread
      2. Configuring the callback on fetch responses for the background thread

      The response handler will collect the fetch responses from the broker and create CompletedFetch, instances, much as is done in Fetcher. The newly introduced FetchUtils will be used for both FetchRequestManager and Fetcher to keep the logic as reusable as possible.

      The foreground logic will decompress the data into a Record, which will then be deserialized into a ConsumerRecord for returning to the user.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue KAFKA-14365 Extract common logic from Fetcher

          • Major - Major loss of function.
          • Resolved
          is part of

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14246 Update threading model for Consumer

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue KAFKA-15184 New consumer internals refactoring and clean up

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15188 Implement more of the remaining PrototypeAsyncConsumer APIs

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15304 CompletableApplicationEvents aren't being completed when the consumer is closing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15508 Method always return the same value ApplicationEventProcessor.java

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request #12814

          Web Link GitHub Pull Request #12816

          Web Link GitHub Pull Request #13495

          Web Link GitHub Pull Request #14305

          Web Link GitHub Pull Request #14359

          Web Link GitHub Pull Request #14406

          Activity

            People

              kirktrue Kirk True
              pnee Philip Nee
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: