Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246

Update threading model for Consumer

    XMLWordPrintableJSON

Details

    Description

      Hi community,

       

      We are refactoring the current KafkaConsumer and making it more asynchronous.  This is the master Jira to track the project's progress; subtasks will be linked to this ticket.  Please review the design document and feel free to use this thread for discussion. 

       

      The design document is here: https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor

       

      The original email thread is here: https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l

       

      I will continue to update the 1pager as reviews and comments come.

       

      Thanks, 

      P

      Attachments

        Issue Links

          Blocked

          Task - A task that needs to be done. KAFKA-14875 Implement Wakeup()

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

          • Major - Major loss of function.
          • Open
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-15316 CommitRequestManager not calling RequestState callbacks

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-15327 Async consumer should commit offsets on close

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-15475 Timeout request might retry forever even if the user API times out in AsyncConsumer

          • Major - Major loss of function.
          • Open

          Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-14274 Introduce FetchRequestManager to integrate fetch into new consumer threading refactor

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-15164 Extract reusable logic from OffsetsForLeaderEpochClient

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-15270 Integration tests for AsyncConsumer simple consume case

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-15317 Fix for async consumer access to committed offsets with multiple consumers

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. KAFKA-15438 Review exception caching logic used for reset/validate positions in async consumer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. KAFKA-14365 Extract common logic from Fetcher

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-14675 Extract metadata-related tasks from Fetcher into MetadataFetcher

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-14758 Extract inner classes from Fetcher for reuse in refactoring

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-14965 Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-14966 Extract reusable logic from OffsetFetcher

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-15081 Implement new consumer offsetsForTimes

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-15115 Implement resetPositions functionality in OffsetsRequestManager

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-15163 Implement validatePositions functionality for new KafkaConsumer

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. KAFKA-15306 Integrate committed offsets logic when updating fetching positions

          • Major - Major loss of function.
          • Resolved
          requires

          Task - A task that needs to be done. KAFKA-15173 Consumer event queues should be bounded

          • Major - Major loss of function.
          • In Progress

          Activity

            People

              pnee Philip Nee
              pnee Philip Nee
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: