[KAFKA-16092] Queues for Kafka

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 4.0.0, 3.8.0, 3.9.0, 4.1.0
• Labels: queues-for-kafka
• Assignee: Andrew Schofield
• Reporter: Andrew Schofield
• Created: 08/Jan/24 17:25
• Updated: 16/Dec/24 22:29

Description

This Jira tracks the development of KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• Add ConsumerGroupPartitionAssignor interface: Resolved
• Add new RPC definitions: Resolved
• Abstract consumer group in coordinator to share functionality with share group: Resolved
• Define skeleton for SharePartitionManager and SharePartition: Resolved
• Create KafkaShareConsumer interface: Resolved
• Add new options for kafka-producer-perf-test.sh: Resolved
• Add exceptions for the new error codes: Resolved
• Implement acquire functionality in SharePartition: Resolved
• Implement share sessions and context: Resolved
• Implement share fetch messages: Resolved
• Add Share Group dynamic and broker configs: Resolved
• Add broker configurations: Resolved
• Define Persister and Share Coordinator RPCs: Resolved
• Implement acknowledge functionality in SharePartition: Resolved
• Integrate persister read API in Partition leader initilization: Resolved
• FindCoordinator v6 initial implementation: Resolved
• Implement share response handling in SharePartitionManager: Resolved
• Implement release acquired records in SharePartitionManager: Resolved
• Implement release acquired records functionality in SharePartition: Resolved
• Implement lock timeout functionality in SharePartition: Resolved
• Implement SharePartitionManager close functionality: Resolved
• MINOR: Remove extra synchronized blocks in SharePartitionManager: Resolved
• Initial code for share-group consumer: Resolved
• Implement updateCacheAndOffsets functionality on LSO movement: Resolved
• Refactor Consumer group and shared classes with Share to modern package: Resolved
• Add ShareGroupHeartbeat API support in GroupCoordinator: Resolved
• SharePartition - Add more unit tests: Resolved
• Add support for share group heartbeat in KafkaApis: Resolved
• Add support for share group describe in KafkaApis: Resolved
• Implement acknowledge API in SharePartitionManager: Resolved
• Implement max delivery count functionality in SharePartition: Resolved
• Support for share-group-metrics in the broker: Resolved
• Retract v6 of ListGroups RPC added in error: Resolved
• Clients : Optimise batching of requests per node in ShareConsumeRequestManager: Resolved
• Add support for share fetch request in KafkaApis: Resolved
• Add support for share acknowledgement request in KafkaApis: Resolved
• Add ShareGroupDescribe API support in GroupCoordinator: Resolved
• Add kafka-console-share-consumer.sh: Resolved
• kafka-share-groups.sh supporting list and describe: Resolved
• Add AdminClient.describeShareGroups and AdminClient.listShareGroups: Resolved
• Broker fixes for smooth concurrent fetches on share partition: Resolved
• Refactor consumer membership managers: Resolved
• Revised share group record schemas: Resolved
• Add SHARE to group.coordinator.rebalance.protocols: Resolved
• Introduce ConsumerRecord.deliveryCount() and remove deprecations: Resolved
• hande piggybacked acknowledgements asynchronously in Share Fetch and properly combine the results of fetch and acknowledge: Resolved
• Add check for whether the requested partition exists or not in Share Fetch request: Resolved
• Add delivery count to kafka-console-share-consumer.sh: Resolved
• Add integration test for ShareGroupDescribe requests: Resolved
• Clients : Revisit result handling in ShareConsumeRequestManager: Resolved
• Initial performance testing fixes: Resolved
• Add integration test for share group list and describe admin calls: Resolved
• Refactor consumer heartbeat managers: Resolved
• Introduce share-coordinator module and add resources: Resolved
• Improve coexistence of consumer groups and share groups: Resolved
• Move common coordinator code from group-coordinator into separate module coordinator-common: Resolved
• Optimise the handler methods in ShareConsumeRequestManager.: Resolved
• Improve kafka-share-groups.sh --describe output for empty groups: Resolved
• Remove tracking member partition epoch for share groups: Resolved
• Fix releasing of session on clients final epoch: Resolved
• Add integration test for Share Group heartbeat: Resolved
• Refactor Share Sesssion Class to Server Module: Resolved
• Implement persister error handling and make write calls async: Resolved
• Make share partition initialization async: Resolved
• Complete pending fetch request on broker shutdown: Resolved
• NOT_LEADER_OR_FOLLOWER with metadata redirection in share group response: Resolved
• Share consumer node latency metrics: Resolved
• Move appropriate classes and interfaces to the new share module: Resolved
• Modify handling of commitSync() in ShareConsumeRequestManager: Resolved
• Introduce a delayed action queue to complete purgatory actions outside purgatory code.: Resolved
• Add integration test for ShareGroupFetch/Acknowledge requests: Resolved
• Introduce a purgatory to deal with share fetch requests that cannot be completed instantaneously: Resolved
• Add config for purge interval for Share Fetch Purgatory: Resolved
• Add support for formatting new records written to offsets topic in kafka-dump-log.sh: Resolved
• Move common fetch related classes from storage to server-common: Resolved
• Share partition acquire() need not return a future: Resolved
• Simplify share partition acquire API: Resolved
• Tighten up wakeup handling for share consumer: Resolved
• Generate ShareAcknowledgeCommitCallbackEvent only when the callback is configured.: Resolved
• Minor tidying of ShareConsumeRequestManager: Resolved
• Move DelayedActionsQueue outside DelayedShareFetch: Resolved
• Add dynamic group configuration for record lock duration: Resolved
• Restructure Share Group Configs: Resolved
• Add integration test for KafkaShareConsumer: Resolved
• Remove unnecessary ignorable annotations from RPC schemas: Resolved
• Add share coordinator implementation: Resolved
• Clients - resolve race condition for SubscriptionState in share group consumer: Resolved
• Move DelayedShareFetchPurgatory declaration to ReplicaManager: Resolved
• Implement formatter for share group state topic records: Resolved
• Changed fetch queue processing to reduce the no. of locking and unlocking activity: Resolved
• Add dynamic share group configurations group.share.heartbeat.interval.ms and group.share.session.timeout.ms: Resolved
• Get rid of fetch queue in SharePartitionManager: Resolved
• Add tests and fix to complete share fetch request on exception from manager: Resolved
• Add capability for max fetch records in share fetch: Resolved
• Integrate partition leader epoch in Share Partition: Resolved
• Resolve reliability issue on the broker post moving delayed share fetch purgatory to ReplicaManager: Resolved
• Align share consumer poll result with record batch boundaries: Resolved
• Add dynamic group configuration for heartbeat interval and session timeout: Resolved
• Handle acknowledgements when subscription changes.: Resolved
• Align string conversion of SharePartitionKey: Resolved
• Implement DefaultShareStatePersister: Resolved
• Add ShareRoundTripWorker for ShareConsumers.: Resolved
• Add minBytes implementation to DelayedShareFetch: Resolved
• Move some purgatory classes from core to share: Resolved
• Verify ShareFetch requests return a completed/erroneous future on purgatory close: Resolved
• Set default value for share.auto.offset.reset in ShareRoundTripWorker.: Resolved
• Console share consumer unable to consume all messages when using multiple brokers: Resolved
• Timed calls to future.get in DefaultStatePersister and test improvements: Resolved
• Remove spurious public constructor for KafkaShareConsumer: Resolved
• Refactor code between SharePartitionManager and DelayedShareFetch for share partition initialization: Resolved
• Error handling when calling replicaManager.getPartitionOrException: Resolved
• Potential issue during tryComplete and onComplete simultaneous calls to access global variables: Resolved
• fetchOffsetMetadata handling for minBytes estimation in both common/uncommon cases of share fetch: Resolved
• Fix bugs in handling piggyback acknowledgements in ShareFetch: Resolved
• Handle response log result error in DelayedShareFetch: Resolved
• Add dynamic group configuration for offset reset: Resolved
• Refactor ShareCoordinator.partitionFor to use SharePartitionKey argument: Resolved
• Add support for formatting records written to share-group state topic in kafka-dump-log.sh: Resolved
• Remove Admin.listShareGroups in favour of Admin.listGroups: Resolved
• `consumer-config` does not work with console-share-consumer: Resolved
• Fix a bug related to updating the callbacks after commitAsync.: Resolved
• Add a log message for KafkaShareConsumer early access: Resolved
• Add support for duration-based offset reset strategy: Resolved
• Remove sharePartition from SharePartitionManager if the partition is deleted or becomes a follower: Resolved
• `SharePartition#maybeInitialize` should complete the future outsize the write lock: Resolved
• Use leader epoch while calling replicaManager.fetchOffsetForTimestamp in ShareFetchUtils: Resolved
• Add error handling in the cases when replicaManager.fetchOffsetForTimestamp throws an error: Resolved
• Create specific classes for share group members and assignments: Resolved
• Investigate missing values for share-coordinator-metrics and share-group-metrics in the broker: Resolved
• the persister methods deleteState, initializeState, and readSummary should not declare exceptions in their method signatures: Resolved
• Enable propagation of the error message when writing state: Resolved
• Null and leaked AcquisitionLockTimerTask causes hanging AcknowledgeRequest and corrupted state of batch: Resolved
• Fix bug in response handler for ShareAcknowledge in clients.: Resolved
• Persist higher leaderEpoch in read state calls to the share coordinator.: Resolved
• Add ShareConsumeBenchWorker for running trogdor workloads.: Resolved
• Empty in memory acknowledgements when a shareAcknowledge request fails: Resolved
• Update nodesWithPendingRequests only when a request successfully builds: Resolved
• Check if nodeToSend is null before sending ShareAcknowledge.: Resolved
• Algorithm to calculate redundant interval of offsets for cleanup: Resolved
• Integrate offset cleanup algorithm with share coordinator: Resolved
• Consumption degrading when using DefaultStatePersister for 1 million records of size 1KB: Resolved
• Define group.version=2: In Progress
• Implement kafka-share-groups.sh --describe --verbose: Resolved
• Add integration test to ShareConsumerTest for share.auto.offset.reset=by_duration: Resolved
• Add "entityType": "topicName" to "SubscribedTopicNames" in ShareGroupHeartbeatRequest.json: Resolved
• Implement registerMetricsForSubscription: In Progress
• Add AdminClient.listShareGroupOffsets: In Progress
• Handle fenced exception while acquiring fetch lock: Open
• Fix testAcknowledgementCommitCallbackInvalidRecordStateException: Open
• Disallow piggyback acknowledgments on initial ShareFetchRequest.: Open
• Fix flaky test: testExplicitAcknowledgementCommitAsync: Reopened
• Console share consumer is not terminated immediately: Open
• Rotate share partitions in share fetch: Open
• Validate compacted topics start offset handling in Share Partition: Open
• Create VerifiableShareConsumer: Open
• Add AdminClient.alterShareGroupOffsets: Open
• Add AdminClient.deleteShareGroupOffsets: Open
• Add AdminClient.deleteShareGroups: Open
• Support for read-committed isolation level: Open
• Create a tool to manipulate share-group state snapshots for recovery purposes: Open
• Improve handling of delivery count: Open
• Periodic snapshots for cold share partitions: Open
• Refactor share coordinator and related classes for better readability and organization: Open
• Refactor AsyncConsumerMetrics so they are appropriate for share consumers: Open
• Prune share state topic metrics: Open
• Optimizing SharePartition locking to improve fetch speed: Open
• Require client-generated member IDs for ShareGroupHeartbeat: Open