Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Hi community,
We are refactoring the current KafkaConsumer and making it more asynchronous. This is the master Jira to track the project's progress; subtasks will be linked to this ticket. Please review the design document and feel free to use this thread for discussion.
The design document is here: https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor
The original email thread is here: https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l
I will continue to update the 1pager as reviews and comments come.
Thanks,
P
Attachments
Attachments
Issue Links
- Blocked
-
KAFKA-14875 Implement Wakeup()
- Resolved
-
KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer
- Open
- incorporates
-
KAFKA-14274 Introduce FetchRequestManager to integrate fetch into new consumer threading refactor
- Resolved
-
KAFKA-15270 Integration tests for AsyncConsumer simple consume case
- Resolved
-
KAFKA-15538 Client support for java regex based subscription
- Resolved
-
KAFKA-15544 Enable existing client integration tests for new protocol
- Resolved
-
KAFKA-15554 Update client state machine to align with protocol sending one assignment at a time
- Resolved
-
KAFKA-14365 Extract common logic from Fetcher
- Resolved
-
KAFKA-14675 Extract metadata-related tasks from Fetcher into MetadataFetcher
- Resolved
-
KAFKA-14724 Port tests in FetcherTest to FetchRequestManagerTest
- Resolved
-
KAFKA-14758 Extract inner classes from Fetcher for reuse in refactoring
- Resolved
-
KAFKA-14965 Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor
- Resolved
-
KAFKA-14966 Extract reusable logic from OffsetFetcher
- Resolved
-
KAFKA-15081 Implement new consumer offsetsForTimes
- Resolved
-
KAFKA-15115 Implement resetPositions functionality in OffsetsRequestManager
- Resolved
-
KAFKA-15163 Implement validatePositions functionality for new KafkaConsumer
- Resolved
-
KAFKA-15164 Extract reusable logic from OffsetsForLeaderEpochClient
- Resolved
-
KAFKA-15306 Integrate committed offsets logic when updating fetching positions
- Resolved
-
KAFKA-15316 CommitRequestManager not calling RequestState callbacks
- Resolved
-
KAFKA-15327 Client consumer should commit offsets on close
- Resolved
-
KAFKA-15455 Add support for OffsetCommit version 9 in consumer
- Resolved
-
KAFKA-15456 Add support for OffsetFetch version 9 in consumer
- Resolved
-
KAFKA-15531 Ensure coordinator node is removed upon disconnection exception
- Resolved
-
KAFKA-15540 Handle heartbeat and revocation when consumer leaves group
- Resolved
-
KAFKA-15317 Fix for async consumer access to committed offsets with multiple consumers
- Resolved
-
KAFKA-15438 Review exception caching logic used for reset/validate positions in async consumer
- Resolved
-
KAFKA-15533 Ensure HeartbeatRequestManager only send out some fields once
- Resolved
-
KAFKA-15539 Client should stop fetching while partitions being revoked
- Closed
-
KAFKA-15515 Remove duplicated integration tests for new consumer
- Closed
-
KAFKA-15175 Assess the use of nio2 asynchronous channel for KafkaConsumer
- Open
- requires
-
KAFKA-15173 Consumer event queues should be bounded
- Open