Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246 Update threading model for Consumer
  3. KAFKA-14274

Introduce FetchRequestManager to integrate fetch into new consumer threading refactor

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The Fetcher class is used internally by the KafkaConsumer to fetch records from the brokers. There is ongoing work to create a new consumer implementation with a significantly refactored threading model. The threading refactor work requires a similarly refactored Fetcher.

      This task is to introduce a new class named FetchRequestManager that will be responsible for:

      1. Formatting fetch requests to the background thread
      2. Configuring the callback on fetch responses for the background thread

      The response handler will collect the fetch responses from the broker and create CompletedFetch, instances, much as is done in Fetcher. The newly introduced FetchUtils will be used for both FetchRequestManager and Fetcher to keep the logic as reusable as possible.

      The foreground logic will decompress the data into a Record, which will then be deserialized into a ConsumerRecord for returning to the user.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kirktrue Kirk True
            pnee Philip Nee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment