Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246

Update threading model for Consumer

    XMLWordPrintableJSON

Details

    Description

      Hi community,

       

      We are refactoring the current KafkaConsumer and making it more asynchronous.  This is the master Jira to track the project's progress; subtasks will be linked to this ticket.  Please review the design document and feel free to use this thread for discussion. 

       

      The design document is here: https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor

       

      The original email thread is here: https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l

       

      I will continue to update the 1pager as reviews and comments come.

       

      Thanks, 

      P

      Attachments

        Issue Links

          Blocked

          Sub-task - The sub-task of the issue KAFKA-14875 Implement Wakeup()

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

          • Major - Major loss of function.
          • Open
          incorporates

          Sub-task - The sub-task of the issue KAFKA-15327 Client consumer should commit offsets on close

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue KAFKA-15455 Add support for OffsetCommit version 9 in consumer

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue KAFKA-15456 Add support for OffsetFetch version 9 in consumer

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue KAFKA-15515 Remove duplicated integration tests for new consumer

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue KAFKA-15538 Client support for java regex based subscription

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Sub-task - The sub-task of the issue KAFKA-15561 Client support for new SubscriptionPattern based subscription

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Sub-task - The sub-task of the issue KAFKA-14274 Introduce FetchRequestManager to integrate fetch into new consumer threading refactor

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15270 Integration tests for AsyncConsumer simple consume case

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15544 Enable existing client integration tests for new protocol

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15554 Update client state machine to align with protocol sending one assignment at a time

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14365 Extract common logic from Fetcher

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14675 Extract metadata-related tasks from Fetcher into MetadataFetcher

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14758 Extract inner classes from Fetcher for reuse in refactoring

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14965 Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-14966 Extract reusable logic from OffsetFetcher

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15081 Implement new consumer offsetsForTimes

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15115 Implement resetPositions functionality in OffsetsRequestManager

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15163 Implement validatePositions functionality for new KafkaConsumer

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15164 Extract reusable logic from OffsetsForLeaderEpochClient

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15306 Integrate committed offsets logic when updating fetching positions

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15316 CommitRequestManager not calling RequestState callbacks

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15531 Ensure coordinator node is removed upon disconnection exception

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15539 Client should stop fetching while partitions being revoked

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15540 Handle heartbeat and revocation when consumer leaves group

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15317 Fix for async consumer access to committed offsets with multiple consumers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15438 Review exception caching logic used for reset/validate positions in async consumer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue KAFKA-15533 Ensure HeartbeatRequestManager only send out some fields once

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

          • Major - Major loss of function.
          • Open
          requires

          Sub-task - The sub-task of the issue KAFKA-15173 Consumer event queues should be bounded

          • Major - Major loss of function.
          • Open
          1.
          		 The background thread should try to process the remaining task until the shutdown timer is expired Sub-task Open Philip Nee
          2.
          		 Timeout request might retry forever even if the user API times out in PrototypeAsyncConsumer Sub-task Open Philip Nee
          3.
          		 Remove duplicated integration tests for new consumer Sub-task Open Unassigned
          4.
          		 DefaultBackgroundThread is running tight loop Sub-task Open Philip Nee
          5.
          		 Document event queueing patterns Sub-task Open Kirk True
          6.
          		 Consumer event queues should be bounded Sub-task Open Kirk True
          7.
          		 Implement ConsumerGroupProtocolVersionResolver to determine consumer group protocol Sub-task Open Kirk True
          8.
          		 Send GroupConsumerHeartbeatRequest on Consumer.close() Sub-task Open Philip Nee
          9.
          		 Evaluate conditions for short circuiting consumer API calls Sub-task Open Philip Nee
          10.
          		 Review committed offset refresh logic Sub-task Open Philip Nee
          11.
          		 Remove NetworkClientDelegate methods isUnavailable, maybeThrowAuthFailure, and tryConnect Sub-task Patch Available Phuc Hong Tran
          12.
          		 Determine if Timer should be used elsewhere in PrototypeAsyncConsumer.updateFetchPositions() Sub-task Open Unassigned
          13.
          		 Improve handling of fetching during metadata updates Sub-task Open Kirk True
          14.
          		 Investigate FetcherTest's/FetchRequestManager's testCompletedFetchRemoval Sub-task Open Kirk True
          15.
          		 Investigate FetcherTest's/FetchRequestManager's duplicate metadata update in assignFromUserNoId Sub-task Open Kirk True
          16.
          		 Investigate FetcherTest's/FetchRequestManager's testFetchingPendingPartitions and testInflightFetchOnPendingPartitions overlap Sub-task Open Kirk True
          17.
          		 Investigate FetcherTest's/FetchRequestManager's testQuotaMetrics Sub-task Open Kirk True
          18.
          		 Investigate FetcherTest's/FetchRequestManager's testFetcherLeadMetric Sub-task Open Kirk True
          19.
          		 Investigate FetcherTest's/FetchRequestManager's testFetchResponseMetrics Sub-task Open Kirk True
          20.
          		 Investigate FetcherTest's/FetchRequestManager's testFetchCompletedBeforeHandlerAdded Sub-task Open Kirk True
          21.
          		 Investigate ConsumerNetworkThreadTest's testPollResultTimer Sub-task Open Philip Nee
          22.
          		 Add tests to verify OffsetFetcherUtils.getOffsetResetTimestamp() Sub-task Open Unassigned
          23.
          		 Investigate ConsumerNetworkThreadTest's testResetPositionsProcessFailureIsIgnored Sub-task Open Philip Nee
          24.
          		 Revoke partitions on Consumer.close() Sub-task Open Philip Nee
          25.
          		 Implement max poll interval Sub-task Patch Available Philip Nee
          26.
          		 Refactor OffsetFetchRequestState Error handling to be more consistent with OffsetCommitRequestState Sub-task Open Lan Ding
          27.
          		 Should ConsumerNetworkThread wrap the exception and notify the polling thread? Sub-task Open Unassigned
          28.
          		 Investigate autocommit retry logic Sub-task Open Unassigned
          29.
          		 Remove excessive use of spy in ConsumerTestBuilder Sub-task Open Unassigned
          30.
          		 Implement ConsumerInterceptor Sub-task Open johndoe
          31.
          		 AsyncKafkaConsumer should retry commits on the application thread instead of auto-retry Sub-task Open Kirk True
          32.
          		 Refactor AsyncKafkaConsumer shutdown Sub-task Open Unassigned

          Activity

            People

              pnee Philip Nee
              pnee Philip Nee
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 528h
                  528h
                  Remaining:
                  Remaining Estimate - 528h
                  528h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified