Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246

Update threading model for Consumer

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Hi community,

       

      We are refactoring the current KafkaConsumer and making it more asynchronous.  This is the master Jira to track the project's progress; subtasks will be linked to this ticket.  Please review the design document and feel free to use this thread for discussion. 

       

      The design document is here: https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor

       

      The original email thread is here: https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l

       

      I will continue to update the 1pager as reviews and comments come.

       

      Thanks, 

      P

      Attachments

        Issue Links

        Blocked

        Sub-task - The sub-task of the issue KAFKA-14875 Implement Wakeup()

        • Major - Major loss of function.
        • Resolved

        Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

        • Major - Major loss of function.
        • Open
        incorporates

        Sub-task - The sub-task of the issue KAFKA-14274 Introduce FetchRequestManager to integrate fetch into new consumer threading refactor

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15270 Integration tests for AsyncConsumer simple consume case

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15538 Client support for java regex based subscription

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15544 Enable existing client integration tests for new protocol

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15554 Update client state machine to align with protocol sending one assignment at a time

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14365 Extract common logic from Fetcher

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14675 Extract metadata-related tasks from Fetcher into MetadataFetcher

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14758 Extract inner classes from Fetcher for reuse in refactoring

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14965 Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-14966 Extract reusable logic from OffsetFetcher

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15081 Implement new consumer offsetsForTimes

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15115 Implement resetPositions functionality in OffsetsRequestManager

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15163 Implement validatePositions functionality for new KafkaConsumer

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15164 Extract reusable logic from OffsetsForLeaderEpochClient

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15306 Integrate committed offsets logic when updating fetching positions

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15316 CommitRequestManager not calling RequestState callbacks

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15327 Client consumer should commit offsets on close

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15455 Add support for OffsetCommit version 9 in consumer

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15456 Add support for OffsetFetch version 9 in consumer

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15531 Ensure coordinator node is removed upon disconnection exception

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15540 Handle heartbeat and revocation when consumer leaves group

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15317 Fix for async consumer access to committed offsets with multiple consumers

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15438 Review exception caching logic used for reset/validate positions in async consumer

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15533 Ensure HeartbeatRequestManager only send out some fields once

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Sub-task - The sub-task of the issue KAFKA-15539 Client should stop fetching while partitions being revoked

        • Major - Major loss of function.
        • Closed

        Test - A new unit, integration or system test. KAFKA-15515 Remove duplicated integration tests for new consumer

        • Major - Major loss of function.
        • Closed

        Wish - General wishlist item. KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer

        • Major - Major loss of function.
        • Open
        requires

        Improvement - An improvement or enhancement to an existing feature or task. KAFKA-15173 Consumer event queues should be bounded

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open
        1.
        		 Propagate client response time when timeout to the request handler Sub-task Resolved Philip Nee   Actions
        2.
        		 Review exception caching logic used for reset/validate positions in async consumer Sub-task Resolved Unassigned   Actions
        3.
        		 CommitRequestManager not calling RequestState callbacks Sub-task Resolved Philip Nee   Actions
        4.
        		 Fix for async consumer access to committed offsets with multiple consumers Sub-task Resolved Unassigned   Actions
        5.
        		 Implement the groupMetadata Consumer API Sub-task Resolved Bruno Cadonna

        0%
        Original Estimate - 504h
        Remaining Estimate - 504h
        		Actions
        6.
        		 Implement more of the remaining PrototypeAsyncConsumer APIs Sub-task Resolved Kirk True   Actions
        7.
        		 New consumer internals refactoring and clean up Sub-task Resolved Kirk True   Actions
        8.
        		 Extract common logic from Fetcher Sub-task Resolved Kirk True   Actions
        9.
        		 Extract inner classes from Fetcher for reuse in refactoring Sub-task Resolved Kirk True   Actions
        10.
        		 Extract metadata-related tasks from Fetcher into MetadataFetcher Sub-task Resolved Kirk True   Actions
        11.
        		 Refactoring for client code to reduce boilerplate Sub-task Resolved Kirk True   Actions
        12.
        		 KafkaConsumer incorrectly passes locally-scoped deserializers to FetchConfig Sub-task Resolved Kirk True

        0%
        Original Estimate - 24h
        Remaining Estimate - 24h
        		Actions
        13.
        		 Port tests in FetcherTest to FetchRequestManagerTest Sub-task Resolved Kirk True   Actions
        14.
        		 Introduce FetchRequestManager to integrate fetch into new consumer threading refactor Sub-task Resolved Kirk True   Actions
        15.
        		 Extract reusable logic from OffsetFetcher Sub-task Resolved Lianet Magrans   Actions
        16.
        		 Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor Sub-task Resolved Lianet Magrans   Actions
        17.
        		 Implement new consumer offsetsForTimes Sub-task Resolved Lianet Magrans   Actions
        18.
        		 Generalize integration tests to change use of KafkaConsumer to Consumer Sub-task Resolved Kirk True   Actions
        19.
        		 Metadata Request Manager and listTopics/partitionsFor API Sub-task Resolved Philip Nee   Actions
        20.
        		 Implement Wakeup() Sub-task Resolved Philip Nee   Actions
        21.
        		 CompletableApplicationEvents aren't being completed when the consumer is closing Sub-task Resolved Philip Nee   Actions
        22.
        		 Integration Tests for the New Consumer Implementation Sub-task Resolved Philip Nee   Actions
        23.
        		 Design & implement support for internal Consumer delegates Sub-task Resolved Kirk True   Actions
        24.
        		 Send GroupConsumerHeartbeatRequest on Consumer.close() Sub-task Resolved Philip Nee   Actions
        25.
        		 OffsetsForTimes validation for negative timestamps in new consumer Sub-task Resolved Lianet Magrans   Actions
        26.
        		 Method always return the same value ApplicationEventProcessor.java Sub-task Resolved Kirk True   Actions
        27.
        		 Ensure wakeups are handled correctly in PrototypeAsyncConsumer.poll() Sub-task Closed Bruno Cadonna   Actions
        28.
        		 Ensure fetch offset and commit offset handler handles both timeout and various error types Sub-task Resolved Philip Nee   Actions
        29.
        		 Investigate auto commit guarantees during Consumer.assign() Sub-task Closed Kirk True   Actions
        30.
        		 Create background thread skeleton for new Consumer threading model Sub-task Resolved Philip Nee   Actions
        31.
        		 Ensure the correct thread is executing the callbacks Sub-task Resolved Philip Nee   Actions
        32.
        		 Extract reusable logic from OffsetsForLeaderEpochClient Sub-task Resolved Lianet Magrans   Actions
        33.
        		 Implement EventHandler interface and DefaultEventHandler for Consumer Sub-task Resolved Philip Nee   Actions
        34.
        		 Implement assign() and assignment() Sub-task Resolved Philip Nee   Actions
        35.
        		 Implement resetPositions functionality in OffsetsRequestManager Sub-task Resolved Lianet Magrans   Actions
        36.
        		 Implement validatePositions functionality for new KafkaConsumer Sub-task Resolved Lianet Magrans   Actions
        37.
        		 Integration tests for AsyncConsumer simple consume case Sub-task Resolved Lianet Magrans   Actions
        38.
        		 Integrate committed offsets logic when updating fetching positions Sub-task Resolved Lianet Magrans   Actions
        39.
        		 Refactor Commit Logic Sub-task Resolved Philip Nee   Actions
        40.
        		 Refactor coordinator code Sub-task Resolved Philip Nee   Actions
        41.
        		 Revoke partitions on Consumer.close() Sub-task Resolved Philip Nee   Actions
        42.
        		 Update last seen epoch during commit Sub-task Resolved Lucas Brutschy   Actions
        43.
        		 Implement max poll interval Sub-task Resolved Philip Nee   Actions
        44.
        		 Restrict Consumer API to be used from one thread Sub-task Resolved Lucas Brutschy   Actions
        45.
        		 Ensure consumer.poll() execute autocommit callback Sub-task Resolved Lucas Brutschy   Actions
        46.
        		 Autocommit during close consistently fails with exception in background thread Sub-task Resolved Philip Nee   Actions
        47.
        		 Migrate AsyncKafkaConsumerTest away from ConsumerTestBuilder Sub-task Resolved Lucas Brutschy   Actions
        48.
        		 AsyncConsumer does not send a poll event to the background thread Sub-task Resolved Philip Nee   Actions
        49.
        		 AsyncKafkaConsumer will get Invalid Request error when trying to rejoin on fenced/unknown member Id Sub-task Resolved Phuc Hong Tran   Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kirktrue Kirk True
            pnee Philip Nee
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 528h
                528h
                Remaining:
                Remaining Estimate - 528h
                528h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Slack

                  Issue deployment