Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14246

Update threading model for Consumer

    XMLWordPrintableJSON

Details

    Description

      Hi community,

       

      We are refactoring the current KafkaConsumer and making it more asynchronous.  This is the master Jira to track the project's progress; subtasks will be linked to this ticket.  Please review the design document and feel free to use this thread for discussion. 

       

      The design document is here: https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor

       

      The original email thread is here: https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l

       

      I will continue to update the 1pager as reviews and comments come.

       

      Thanks, 

      P

      Attachments

        Issue Links

          1.
          Propagate client response time when timeout to the request handler Sub-task Resolved Philip Nee  
          2.
          Review exception caching logic used for reset/validate positions in async consumer Sub-task Resolved Unassigned  
          3.
          CommitRequestManager not calling RequestState callbacks Sub-task Resolved Philip Nee  
          4.
          Fix for async consumer access to committed offsets with multiple consumers Sub-task Resolved Unassigned  
          5.
          Implement the groupMetadata Consumer API Sub-task Resolved Bruno Cadonna

          0%

          Original Estimate - 504h
          Remaining Estimate - 504h
          6.
          Implement more of the remaining PrototypeAsyncConsumer APIs Sub-task Resolved Kirk True  
          7.
          New consumer internals refactoring and clean up Sub-task Resolved Kirk True  
          8.
          Extract common logic from Fetcher Sub-task Resolved Kirk True  
          9.
          Extract inner classes from Fetcher for reuse in refactoring Sub-task Resolved Kirk True  
          10.
          Extract metadata-related tasks from Fetcher into MetadataFetcher Sub-task Resolved Kirk True  
          11.
          Refactoring for client code to reduce boilerplate Sub-task Resolved Kirk True  
          12.
          KafkaConsumer incorrectly passes locally-scoped deserializers to FetchConfig Sub-task Resolved Kirk True

          0%

          Original Estimate - 24h
          Remaining Estimate - 24h
          13.
          Port tests in FetcherTest to FetchRequestManagerTest Sub-task Resolved Kirk True  
          14.
          Introduce FetchRequestManager to integrate fetch into new consumer threading refactor Sub-task Resolved Kirk True  
          15.
          Extract reusable logic from OffsetFetcher Sub-task Resolved Lianet Magrans  
          16.
          Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor Sub-task Resolved Lianet Magrans  
          17.
          Implement new consumer offsetsForTimes Sub-task Resolved Lianet Magrans  
          18.
          Generalize integration tests to change use of KafkaConsumer to Consumer Sub-task Resolved Kirk True  
          19.
          Metadata Request Manager and listTopics/partitionsFor API Sub-task Resolved Philip Nee  
          20.
          Implement Wakeup() Sub-task Resolved Philip Nee  
          21.
          CompletableApplicationEvents aren't being completed when the consumer is closing Sub-task Resolved Philip Nee  
          22.
          Integration Tests for the New Consumer Implementation Sub-task Resolved Philip Nee  
          23.
          Design & implement support for internal Consumer delegates Sub-task Resolved Kirk True  
          24.
          Send GroupConsumerHeartbeatRequest on Consumer.close() Sub-task Resolved Philip Nee  
          25.
          OffsetsForTimes validation for negative timestamps in new consumer Sub-task Resolved Lianet Magrans  
          26.
          Method always return the same value ApplicationEventProcessor.java Sub-task Resolved Kirk True  
          27.
          Ensure wakeups are handled correctly in PrototypeAsyncConsumer.poll() Sub-task Closed Bruno Cadonna  
          28.
          Ensure fetch offset and commit offset handler handles both timeout and various error types Sub-task Resolved Philip Nee  
          29.
          Investigate auto commit guarantees during Consumer.assign() Sub-task Closed Kirk True  
          30.
          Create background thread skeleton for new Consumer threading model Sub-task Resolved Philip Nee  
          31.
          Ensure the correct thread is executing the callbacks Sub-task Resolved Philip Nee  
          32.
          Extract reusable logic from OffsetsForLeaderEpochClient Sub-task Resolved Lianet Magrans  
          33.
          Implement EventHandler interface and DefaultEventHandler for Consumer Sub-task Resolved Philip Nee  
          34.
          Implement assign() and assignment() Sub-task Resolved Philip Nee  
          35.
          Implement resetPositions functionality in OffsetsRequestManager Sub-task Resolved Lianet Magrans  
          36.
          Implement validatePositions functionality for new KafkaConsumer Sub-task Resolved Lianet Magrans  
          37.
          Integration tests for AsyncConsumer simple consume case Sub-task Resolved Lianet Magrans  
          38.
          Integrate committed offsets logic when updating fetching positions Sub-task Resolved Lianet Magrans  
          39.
          Refactor Commit Logic Sub-task Resolved Philip Nee  
          40.
          Refactor coordinator code Sub-task Resolved Philip Nee  
          41.
          Revoke partitions on Consumer.close() Sub-task Resolved Philip Nee  
          42.
          Update last seen epoch during commit Sub-task Resolved Lucas Brutschy  
          43.
          Implement max poll interval Sub-task Resolved Philip Nee  
          44.
          Restrict Consumer API to be used from one thread Sub-task Resolved Lucas Brutschy  
          45.
          Ensure consumer.poll() execute autocommit callback Sub-task Resolved Lucas Brutschy  
          46.
          Autocommit during close consistently fails with exception in background thread Sub-task Resolved Philip Nee  
          47.
          Migrate AsyncKafkaConsumerTest away from ConsumerTestBuilder Sub-task Resolved Lucas Brutschy  
          48.
          AsyncConsumer does not send a poll event to the background thread Sub-task Resolved Philip Nee  
          49.
          AsyncKafkaConsumer will get Invalid Request error when trying to rejoin on fenced/unknown member Id Sub-task Resolved Phuc Hong Tran  

          Activity

            People

              kirktrue Kirk True
              pnee Philip Nee
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 528h
                  528h
                  Remaining:
                  Remaining Estimate - 528h
                  528h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified