Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14048

The Next Generation of the Consumer Rebalance Protocol

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • None
    • 4.0.0
    • None
    • None

    Attachments

      Issue Links

        1.
        Automated protocol should support namespace Sub-task Open David Jacot  
        2.
        Add new `group-coordinator` module Sub-task Resolved David Jacot  
        3.
        Introduce `GroupCoordinator` interface Sub-task Resolved David Jacot  
        4.
        Add ConsumerGroupHeartbeat API Sub-task Resolved David Jacot  
        5.
        The Kafka protocol should support nullable structs Sub-task Resolved David Jacot  
        6.
        New Group Coordinator State Machine Sub-task Resolved David Jacot  
        7.
        Implement OffsetFetch/OffsetCommit APIs Sub-task Resolved David Jacot  
        8.
        Implement JoinGroup/SyncGroup APIs Sub-task Resolved Jeff Kim  
        9.
        Implement Heartbeat API Sub-task Resolved Jeff Kim  
        10.
        Implement LeaveGroup API Sub-task Resolved Jeff Kim  
        11.
        Implement ListGroups Sub-task Resolved HaiyuanZhao  
        12.
        Implement DescribeGroups API Sub-task Resolved Dongnuo Lyu  
        13.
        Implement TnxOffsetCommit API Sub-task Resolved David Jacot  
        14.
        Implement DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        15.
        Add ConsumerGroupDescribe API Sub-task Resolved Max Riedel  
        16.
        Extend DescribeConfigs API to support group configs Sub-task Resolved Lan Ding  
        17.
        Extend AlterIncrementalConfigs API to support group config Sub-task Resolved Lan Ding  
        18.
        Add broker side PartitionAssignor interface Sub-task Resolved David Jacot  
        19.
        Implement range broker side assignor Sub-task Resolved Ritika Reddy  
        20.
        Implement optimized uniform broker side assignor Sub-task Resolved Ritika Reddy  
        21.
        Implement static membeship Sub-task Resolved Sagar Rao  
        22.
        Implement regex subscriptions Sub-task Resolved Lianet Magrans  
        23.
        Rebalance on topic/partition metadata changes Sub-task Resolved David Jacot  
        24.
        Add metrics to the new coordinator Sub-task Resolved Jeff Kim  
        25.
        Add high watermark listener to Partition/Log layers Sub-task Resolved David Jacot  
        26.
        Move __consumer_offsets records from core to group-coordinator Sub-task Resolved David Jacot  
        27.
        Add new __consumer_offsets records Sub-task Resolved David Jacot  
        28.
        More broker side partition assignor to common Sub-task Resolved David Jacot  
        29.
        Extend server side assignor to support rack aware replica placement Sub-task Resolved Ritika Reddy  
        30.
        Add ConsumerGroupHeartbeat API to AuthorizerIntegrationTest Sub-task Resolved Lan Ding  
        31.
        Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic Sub-task Resolved Alexandre Dupriez  
        32.
        Propagate Topic Ids to the Group Coordinator during Offsets Commit Sub-task Resolved Alexandre Dupriez  
        33.
        Propagate Topic Ids to the Group Coordinator for Offset Fetch Sub-task Resolved Alexandre Dupriez  
        34.
        Refactor ApiMessageAndVersion/Record Sub-task Resolved David Jacot  
        35.
        Validate that all partitions are assigned in TargetAssignmentBuilder Sub-task Resolved Unassigned  
        36.
        Implement Group/Offset expiration Sub-task Resolved Jeff Kim  
        37.
        CoordinatorPartitionWriter should reuse buffer Sub-task Resolved David Jacot  
        38.
        Consolidate all the group coordinator configs Sub-task Resolved Unassigned  
        39.
        Define constants for record types Sub-task Resolved Unassigned  
        40.
        Implement write operation timeout Sub-task Resolved Sagar Rao  
        41.
        CoordinatorContext should be protected by a lock Sub-task Resolved David Jacot  
        42.
        Consider replacing Subscription Metadata by a hash Sub-task Resolved David Jacot  
        43.
        Add support for OffsetCommit version 9 in admin client Sub-task Resolved Sagar Rao  
        44.
        Add support for OffsetCommit version 9 in consumer Sub-task Resolved Lianet Magrans  
        45.
        Add support for OffsetFetch version 9 in consumer Sub-task Resolved Lianet Magrans  
        46.
        Add support for OffsetFetch version 9 in admin Sub-task Resolved Sagar Rao  
        47.
        Add group type filter to ListGroups API Sub-task Resolved Ritika Reddy  
        48.
        Add integration test for the ListGroup API Sub-task Resolved Dongnuo Lyu  
        49.
        Add group type filter to the admin client Sub-task Resolved Ritika Reddy  
        50.
        Implement general uniform broker side assignor Sub-task Resolved Ritika Reddy  
        51.
        Handle heartbeat and revocation when consumer leaves group Sub-task Resolved Unassigned  
        52.
        Client consumer should commit offsets on close Sub-task Resolved Philip Nee  
        53.
        Send HB request right after reconciliation completes Sub-task Resolved Unassigned  
        54.
        Ensure HeartbeatRequestManager only send out some fields once Sub-task Resolved Philip Nee  
        55.
        Ensure coordinator node is removed upon disconnection exception Sub-task Resolved Philip Nee  
        56.
        Enable existing client integration tests for new protocol Sub-task Resolved Andrew Schofield  
        57.
        Implement client support for selecting KIP-848 server-side assignor Sub-task Resolved Lucas Brutschy  
        58.
        Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC Sub-task Resolved Kirk True  
        59.
        Implement event plumbing for ConsumerRebalanceListener callbacks Sub-task Resolved Kirk True  
        60.
        Implement consumer group membership state machine Sub-task Resolved Lianet Magrans  
        61.
        Client should stop fetching while partitions being revoked Sub-task Closed Lianet Magrans  
        62.
        Update client state machine to align with protocol sending one assignment at a time Sub-task Resolved Lianet Magrans  
        63.
        Implement auto-commit on partition assignment revocation Sub-task Closed Lianet Magrans  
        64.
        Update states and transitions for membership manager state machine Sub-task Resolved Lianet Magrans  
        65.
        Run System Tests for Old protocol in the New Coordinator Sub-task Resolved Ritika Reddy

        100%

        Original Estimate - 672h
        Time Not Required
        66.
        Member does not need to always try to join a group when a groupId is configured Sub-task Closed Philip Nee  
        67.
        Add integration tests for DescribeGroups API, DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        68.
        Add histogram metrics to GroupCoordinatorRuntimeMetrics Sub-task Resolved Jeff Kim  
        69.
        Refactor ConsumerRebalanceListener invocation for reuse Sub-task Resolved Kirk True  
        70.
        Do not send new heartbeat request while another one in-flight Sub-task Closed Philip Nee  
        71.
        Improve unloading logging Sub-task Resolved Ritika Reddy  
        72.
        Fix the different behavior in error handling between the old and new group coordinator Sub-task Resolved Dongnuo Lyu  
        73.
        Client support for new consumer configs Sub-task Resolved Philip Nee  
        74.
        New integration tests to have full coverage for preview Sub-task Closed Kirk True  
        75.
        Add integration tests for Heartbeat API and GroupLeave API Sub-task Resolved Dongnuo Lyu  
        76.
        Migrate existing integration tests to run old protocol in new coordinator Sub-task Resolved Dongnuo Lyu  
        77.
        Group Coordinator should not deliver new assignment before previous one is acknowledged Sub-task Resolved David Jacot  
        78.
        Implement listTopics() and partitionFor() for the AsyncKafkaConsumer Sub-task Resolved Andrew Schofield  
        79.
        Correct initialization of ConsumerGroupHeartbeat by client Sub-task Closed Andrew Schofield  
        80.
        Correct handling of KafkaConsumer.committed for new consumer Sub-task Resolved Andrew Schofield  
        81.
        Fix ListGroups API when runtime partition size is zero Sub-task Resolved Dongnuo Lyu  
        82.
        Add integration tests for JoinGroup API and SyncGroup API Sub-task Resolved Dongnuo Lyu  
        83.
        Move new group coordinator metrics from Yammer to Metrics Sub-task Resolved Jeff Kim  
        84.
        Make General Assignor's rebalance incremental Sub-task Resolved Unassigned  
        85.
        New group coordinator needs to generate snapshots while loading Sub-task Resolved Jeff Kim  
        86.
        Refactor polling delays Sub-task Resolved Andrew Schofield  
        87.
        update Group size only when groups size changes Sub-task Resolved Jeff Kim  
        88.
        Move GenericGroup state metrics to `GroupCoordinatorMetricsShard` Sub-task Resolved Jeff Kim  
        89.
        Upgrade existing generic group to consumer group Sub-task Resolved David Jacot  
        90.
        Ensure fairness in the uniform assignor Sub-task Resolved David Jacot  
        91.
        new group coordinator should check if partition goes offline during load Sub-task Resolved Jeff Kim  
        92.
        Add `group.coordinator.rebalance.protocols` and publish all new configs Sub-task Resolved David Jacot  
        93.
        Rename `Generic` to `Classic` Sub-task Resolved David Jacot  
        94.
        Trigger client reconciliation based on manager poll Sub-task Resolved Lucas Brutschy  
        95.
        Review client retry logic of OffsetFetch and OffsetCommit responses Sub-task Resolved Lianet Magrans  
        96.
        Client support for java regex based subscription Sub-task Resolved Phuc Hong Tran  
        97.
        Group commit/callbacks triggering logic Sub-task Open Lianet Magrans  
        98.
        Review consumer leave group request best effort and response handling Sub-task Resolved Unassigned  
        99.
        Review minimal effort approach on consumer last heartbeat on unsubscribe Sub-task Open Lianet Magrans  
        100.
        Review consumer onPartitionsAssigned called with empty partitions Sub-task Open Lianet Magrans  
        101.
        Topic ID integration in consumer subscription state Sub-task Closed Lianet Magrans  
        102.
        Update list group state type filter to include the states for the new consumer group type Sub-task Resolved Lan Ding  
        103.
        Handle timeouts for AsyncKafkaConsumer.commitSync Sub-task Resolved Unassigned  
        104.
        AsyncKafkaConsumer: Always await async commit callbacks in commitSync and close Sub-task Resolved Lucas Brutschy  
        105.
        group size counters do not reflect the actual sizes when operations fail Sub-task Resolved Dongnuo Lyu  
        106.
        Ensure consumer does not start fetching from added partitions until onPartitionsAssigned completes Sub-task Resolved Lianet Magrans  
        107.
        Review client errors thrown on OffsetFetch and OffsetCommit failures Sub-task Resolved Lianet Magrans  
        108.
        Add Integration test for checking if the correct assignor is chosen Sub-task Resolved Unassigned  
        109.
        Coordinator unloading fails when replica is deleted Sub-task Resolved David Jacot  
        110.
        Partition is assigned to two members at the same time Sub-task Resolved David Jacot  
        111.
        Implement GroupMetadataManager#onUnloaded Sub-task Resolved Jeff Kim  
        112.
        System test failing for new consumer on endOffsets with negative timestamps Sub-task Resolved Philip Nee  
        113.
        Consumer invalid transition on expired poll interval Sub-task Resolved Lianet Magrans  
        114.
        Implement GroupCoordinator.onPartitionsDeleted Sub-task Resolved David Jacot  
        115.
        Fix client reconciliation of same assignment received in different epochs Sub-task Resolved Lucas Brutschy  
        116.
        Extend admin to support ConsumerGroupDescribe API Sub-task Resolved David Jacot  
        117.
        Member should send full heartbeat when rejoining Sub-task Resolved Quoc Phong Dang  
        118.
        KafkaConsumer.groupMetadata() should be correct when first records are returned Sub-task Closed Bruno Cadonna  
        119.
        Consumer does not rejoin after fenced on delayed revocation Sub-task Resolved Lianet Magrans  
        120.
        Fix handling of deleted topic when auto-committing before revocation Sub-task Closed Bruno Cadonna  
        121.
        Console consumer fails with `IllegalStateException` Sub-task Closed Bruno Cadonna  
        122.
        Move code style exceptions from suppressions.xml to the code Sub-task Open Unassigned  
        123.
        Improve reconciliation state machine Sub-task Resolved David Jacot  
        124.
        Fenced member should not send heartbeats while waiting for onPartitionsLost to complete Sub-task Resolved Lianet Magrans  
        125.
        Stale member should trigger onPartitionsLost when leaving group Sub-task Resolved Lianet Magrans  
        126.
        Add group protocol migration enabling config Sub-task Resolved Dongnuo Lyu  
        127.
        Ensure user callbacks exceptions are propagated to the user on consumer poll Sub-task Resolved Lucas Brutschy  
        128.
        Classic group error responses should contain a no generation id and an empty members list Sub-task Resolved Jeff Kim  
        129.
        Review fenced member unsubscribe/subscribe callbacks interaction Sub-task Open Quoc Phong Dang  
        130.
        GroupCoordinatorService logger is not configured Sub-task Resolved Jeff Kim  
        131.
        fix EventAccumulator thread idle ratio metric Sub-task Resolved Jeff Kim  
        132.
        Offline group protocol migration Sub-task Resolved Dongnuo Lyu  
        133.
        Add event rate in GroupCoordinatorRuntimeMetrics Sub-task Resolved Jeff Kim  
        134.
        Offline protocol migration integration tests Sub-task Resolved Dongnuo Lyu  
        135.
        Full ConsumerGroupHeartbeat response must be sent when full request is received Sub-task Resolved David Jacot  
        136.
        High watermark updates should have a higher priority Sub-task Resolved David Jacot  
        137.
        Fix logic for discarding reconciliation if member rejoined Sub-task Resolved Lianet Magrans  
        138.
        Coordinator flush time and event purgatory time metrics Sub-task Resolved Jeff Kim  
        139.
        Online upgrade triggering and group type conversion Sub-task Resolved Dongnuo Lyu  
        140.
        kafka-dump-log --offsets-decoder should support new records Sub-task Resolved David Jacot  
        141.
        getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException Sub-task Resolved David Jacot  
        142.
        Online downgrade triggering and group type conversion Sub-task Resolved Dongnuo Lyu  
        143.
        Add JMH Benchmarks for assignor performance testing Sub-task Resolved Ritika Reddy  
        144.
        Target Assignment Format Change Sub-task Resolved Ritika Reddy  
        145.
        Store subscription model for consumer group in group state Sub-task Resolved Ritika Reddy  
        146.
        JoinGroup API for upgrading ConsumerGroup Sub-task Resolved Dongnuo Lyu  
        147.
        Reverse Lookup Partition to Member in Assignors Sub-task Resolved Ritika Reddy  
        148.
        Uuid to String for subscribed topic names in assignment spec Sub-task Resolved Jeff Kim  
        149.
        CoordinatorRuntime write timer tasks should be cancelled once HWM advances Sub-task Resolved Jeff Kim  
        150.
        Re-add EventAccumulator.take(timeout) Sub-task Resolved Jeff Kim  
        151.
        Optimize toTopicPartitions with ConsumerProtocolSubscription Sub-task Resolved Dongnuo Lyu  
        152.
        Adjust classicGroupJoinToConsumerGroup to add subscription model Sub-task Resolved Dongnuo Lyu  
        153.
        Remove rack aware code in assignors temporarily due to performance Sub-task Resolved Ritika Reddy  
        154.
        SyncGroup API for upgrading ConsumerGroup Sub-task Resolved Dongnuo Lyu  
        155.
        Coalesce records into bigger batches Sub-task Resolved David Jacot  
        156.
        Heartbeat API for upgrading ConsumerGroup Sub-task Resolved Dongnuo Lyu  
        157.
        Create a new interface to store member metadata Sub-task Resolved Ritika Reddy  
        158.
        CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit Sub-task Resolved Jeff Kim  
        159.
        LeaveGroup API for upgrading ConsumerGroup Sub-task Resolved Dongnuo Lyu  
        160.
        Introduce `group.version` feature flag Sub-task Resolved David Jacot  
        161.
        Copy on write in the Optimized Uniform Assignor Sub-task Resolved David Jacot  
        162.
        UniformHeterogeneousAssignmentBuilder throws NPE when member has not subscriptions Sub-task Resolved David Jacot  
        163.
        Range assignor doesn't co-partition with stickiness Sub-task Resolved Ritika Reddy  
        164.
        Fix caught-up condition Sub-task Resolved David Jacot  
        165.
        Online protocol migration integration tests Sub-task Resolved Dongnuo Lyu  
        166.
        Extend CoordinatorRuntime to support non-atomic writes Sub-task Resolved David Jacot  
        167.
        Static member using new protocol should always replace the one using the old protocol Sub-task Resolved Dongnuo Lyu  
        168.
        System tests for protocol migration Sub-task Resolved Dongnuo Lyu  
        169.
        Soften the validation when replaying tombstones Sub-task Resolved David Jacot  
        170.
        Validate and maybe trigger downgrade after static member replacement Sub-task Resolved Dongnuo Lyu  
        171.
        Use the new group coordinator by default in 4.0 Sub-task Resolved David Jacot  
        172.
        Re-introduce `group.version` feature flag Sub-task Resolved David Jacot  
        173.
        Delete kafka.common.OffsetAndMetadata Sub-task Resolved Mickael Maison  
        174.
        Factor out common setup from server-side assignor benchmarks Sub-task Resolved Sean Quah  
        175.
        Add heterogeneous configuration to TargetAssignmentBuilderBenchmark Sub-task Resolved Sean Quah  
        176.
        Revert #17219 Sub-task Resolved David Jacot  
        177.
        Remove partitionRacks from TopicMetadata Sub-task Resolved PoAn Yang  
        178.
        Reduce logging verbosity on ConsumerGroupHeartbeat path Sub-task Resolved David Jacot  
        179.
        Rework UniformHeterogeneousAssignor to improve performance Sub-task Resolved Sean Quah  
        180.
        Protocol upgrade should allow empty member assignment in group conversion Sub-task Resolved Dongnuo Lyu  
        181.
        Move static request validation from GroupMetadataManager to GroupCoordinatorService Sub-task Open Unassigned  
        182.
        Reject the upgrade if the member assignment has non empty userData Sub-task Resolved Sean Quah  
        183.
        Admin LeaveGroup fails for static consumer group member Sub-task Resolved David Jacot  
        184.
        Maybe downgrade the consumer group if a consumer member leaves the group though admin client Sub-task Open Sean Quah  
        185.
        Mark KIP-848's API & Records as stable Sub-task Resolved David Jacot  
        186.
        Handle hdr histogram exceptions gracefully Sub-task Resolved Jeff Kim  
        187.
        Admin LeaveGroup should allow removing member using consumer protocol by member id Sub-task Resolved David Jacot  
        188.
        Add ConsumerGroupRegularExpression record to dump-log tool Sub-task Resolved David Jacot  
        189.
        Add more logs to the migration path Sub-task Open Dongnuo Lyu  
        190.
        Ensure that assignors are at the right place Sub-task Open David Jacot  
        191.
        Update upgrade notes for 4.0 Sub-task Open David Jacot  

        Activity

          People

            dajac David Jacot
            dajac David Jacot
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 672h
                672h
                Remaining:
                0h
                Logged:
                Time Not Required
                2m