Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14048

The Next Generation of the Consumer Rebalance Protocol

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Attachments

      Issue Links

        1.
        Automated protocol should support namespace Sub-task Open David Jacot  
        2.
        Add new `group-coordinator` module Sub-task Resolved David Jacot  
        3.
        Introduce `GroupCoordinator` interface Sub-task Resolved David Jacot  
        4.
        Add ConsumerGroupHeartbeat API Sub-task Resolved David Jacot  
        5.
        The Kafka protocol should support nullable structs Sub-task Resolved David Jacot  
        6.
        New Group Coordinator State Machine Sub-task Resolved David Jacot  
        7.
        Implement OffsetFetch/OffsetCommit APIs Sub-task Resolved David Jacot  
        8.
        Implement JoinGroup/SyncGroup APIs Sub-task Resolved Jeff Kim  
        9.
        Implement Heartbeat API Sub-task Resolved Jeff Kim  
        10.
        Implement LeaveGroup API Sub-task Resolved Jeff Kim  
        11.
        Implement ListGroups Sub-task Resolved HaiyuanZhao  
        12.
        Implement DescribeGroups API Sub-task Resolved Dongnuo Lyu  
        13.
        Implement TnxOffsetCommit API Sub-task Resolved David Jacot  
        14.
        Implement DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        15.
        Add ConsumerGroupPrepareAssignment API Sub-task Open Phuc Hong Tran  
        16.
        Add ConsumerGroupInstallAssignment API Sub-task Open Phuc Hong Tran  
        17.
        Add ConsumerGroupDescribe API Sub-task In Progress Max Riedel  
        18.
        Extend DescribeConfigs API to support group configs Sub-task Open Jimmy Wang  
        19.
        Extend AlterIncrementalConfigs API to support group config Sub-task In Progress Lan Ding  
        20.
        Add broker side PartitionAssignor interface Sub-task Resolved David Jacot  
        21.
        Implement range broker side assignor Sub-task Resolved Ritika Reddy  
        22.
        Implement optimized uniform broker side assignor Sub-task Resolved Ritika Reddy  
        23.
        Implement static membeship Sub-task Resolved Sagar Rao  
        24.
        Implement regex subscriptions Sub-task In Progress Jimmy Wang  
        25.
        Rebalance on topic/partition metadata changes Sub-task Resolved David Jacot  
        26.
        Add metrics to the new coordinator Sub-task Resolved Jeff Kim  
        27.
        Add high watermark listener to Partition/Log layers Sub-task Resolved David Jacot  
        28.
        Move __consumer_offsets records from core to group-coordinator Sub-task Resolved David Jacot  
        29.
        Add new __consumer_offsets records Sub-task Resolved David Jacot  
        30.
        OffsetCommit API Version 9 Sub-task Patch Available Alexandre Dupriez  
        31.
        OffsetFetch API Version 9 Sub-task In Progress Alexandre Dupriez  
        32.
        More broker side partition assignor to common Sub-task Open Unassigned  
        33.
        Extend server side assignor to support rack aware replica placement Sub-task Resolved Ritika Reddy  
        34.
        Add ConsumerGroupHeartbeat API to AuthorizerIntegrationTest Sub-task Open Unassigned  
        35.
        Add support of topic id for OffsetCommitRequests in CommitRequestManager Sub-task Open Alexandre Dupriez  
        36.
        Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic Sub-task Open Alexandre Dupriez  
        37.
        Propagate Topic Ids to the Group Coordinator during Offsets Commit Sub-task Resolved Alexandre Dupriez  
        38.
        Propagate Topic Ids to the Group Coordinator for Offset Fetch Sub-task Open Alexandre Dupriez  
        39.
        Refactor ApiMessageAndVersion/Record Sub-task Open David Jacot  
        40.
        Validate that all partitions are assigned in TargetAssignmentBuilder Sub-task Open Unassigned  
        41.
        Implement Group/Offset expiration Sub-task Resolved Jeff Kim  
        42.
        CoordinatorPartitionWriter should reuse buffer Sub-task Resolved David Jacot  
        43.
        Consolidate all the group coordinator configs Sub-task Open Unassigned  
        44.
        Define constants for record types Sub-task Open Unassigned  
        45.
        Implement write operation timeout Sub-task Resolved Sagar Rao  
        46.
        CoordinatorContext should be protected by a lock Sub-task Resolved David Jacot  
        47.
        Consider replacing Subscription Metadata by a hash Sub-task Open David Jacot  
        48.
        Add support for OffsetCommit version 9 in admin client Sub-task Resolved Sagar Rao  
        49.
        Add support for OffsetCommit version 9 in consumer Sub-task Resolved Lianet Magrans  
        50.
        Add support for OffsetFetch version 9 in consumer Sub-task Resolved Lianet Magrans  
        51.
        Add support for OffsetFetch version 9 in admin Sub-task Resolved Sagar Rao  
        52.
        Add group type filter to ListGroups API Sub-task Resolved Ritika Reddy  
        53.
        Add integration test for the ListGroup API Sub-task Resolved Dongnuo Lyu  
        54.
        Add group type filter to the admin client Sub-task In Progress Ritika Reddy  
        55.
        Implement general uniform broker side assignor Sub-task Resolved Ritika Reddy  
        56.
        Handle heartbeat and revocation when consumer leaves group Sub-task Resolved Unassigned  
        57.
        Client consumer should commit offsets on close Sub-task Resolved Philip Nee  
        58.
        Send HB request right after reconciliation completes Sub-task Resolved Unassigned  
        59.
        Ensure HeartbeatRequestManager only send out some fields once Sub-task Resolved Philip Nee  
        60.
        Ensure coordinator node is removed upon disconnection exception Sub-task Resolved Philip Nee  
        61.
        Enable existing client integration tests for new protocol Sub-task Resolved Andrew Schofield  
        62.
        Implement client support for selecting KIP-848 server-side assignor Sub-task Resolved Lucas Brutschy  
        63.
        Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC Sub-task Resolved Kirk True  
        64.
        Implement event plumbing for ConsumerRebalanceListener callbacks Sub-task Resolved Kirk True  
        65.
        Implement consumer group membership state machine Sub-task Resolved Lianet Magrans  
        66.
        Client should stop fetching while partitions being revoked Sub-task Closed Lianet Magrans  
        67.
        Update client state machine to align with protocol sending one assignment at a time Sub-task Resolved Lianet Magrans  
        68.
        Implement auto-commit on partition assignment revocation Sub-task Closed Lianet Magrans  
        69.
        Update states and transitions for membership manager state machine Sub-task Resolved Lianet Magrans  
        70.
        Run System Tests for Old protocol in the New Coordinator Sub-task In Progress Ritika Reddy

        0%

        Original Estimate - 672h
        Remaining Estimate - 672h
        71.
        Member does not need to always try to join a group when a groupId is configured Sub-task Closed Philip Nee  
        72.
        Add integration tests for DescribeGroups API, DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        73.
        Add histogram metrics to GroupCoordinatorRuntimeMetrics Sub-task Open Unassigned  
        74.
        Refactor ConsumerRebalanceListener invocation for reuse Sub-task Resolved Kirk True  
        75.
        Do not send new heartbeat request while another one in-flight Sub-task Closed Philip Nee  
        76.
        Improve unloading logging Sub-task Resolved Ritika Reddy  
        77.
        Fix the different behavior in error handling between the old and new group coordinator Sub-task Resolved Dongnuo Lyu  
        78.
        Client support for new consumer configs Sub-task Resolved Philip Nee  
        79.
        New integration tests to have full coverage for preview Sub-task Closed Kirk True  
        80.
        Add integration tests for Heartbeat API and GroupLeave API Sub-task Resolved Dongnuo Lyu  
        81.
        Migrate existing integration tests to run old protocol in new coordinator Sub-task Open Dongnuo Lyu  
        82.
        Group Coordinator should not deliver new assignment before previous one is acknowledged Sub-task Open David Jacot  
        83.
        Implement listTopics() and partitionFor() for the AsyncKafkaConsumer Sub-task Resolved Andrew Schofield  
        84.
        Correct initialization of ConsumerGroupHeartbeat by client Sub-task Closed Andrew Schofield  
        85.
        Correct handling of KafkaConsumer.committed for new consumer Sub-task Resolved Andrew Schofield  
        86.
        Fix ListGroups API when runtime partition size is zero Sub-task Resolved Dongnuo Lyu  
        87.
        Add integration tests for JoinGroup API and SyncGroup API Sub-task Resolved Dongnuo Lyu  
        88.
        Move new group coordinator metrics from Yammer to Metrics Sub-task Resolved Jeff Kim  
        89.
        Make General Assignor's rebalance incremental Sub-task Open Unassigned  
        90.
        New group coordinator needs to generate snapshots while loading Sub-task Resolved Jeff Kim  
        91.
        Refactor polling delays Sub-task Resolved Andrew Schofield  
        92.
        update Group size only when groups size changes Sub-task Resolved Jeff Kim  
        93.
        Move GenericGroup state metrics to `GroupCoordinatorMetricsShard` Sub-task Resolved Jeff Kim  
        94.
        Upgrade existing generic group to consumer group Sub-task Open David Jacot  
        95.
        Ensure fairness in the uniform assignor Sub-task Open Ritika Reddy  
        96.
        new group coordinator should check if partition goes offline during load Sub-task Resolved Jeff Kim  
        97.
        Add `group.coordinator.rebalance.protocols` and publish all new configs Sub-task Resolved David Jacot  
        98.
        Rename `Generic` to `Classic` Sub-task Resolved David Jacot  
        99.
        Trigger client reconciliation based on manager poll Sub-task Resolved Lucas Brutschy  
        100.
        Client support for OffsetFetch and OffsetCommit with topic ID Sub-task Open Lianet Magrans  
        101.
        Review client retry logic of OffsetFetch and OffsetCommit responses Sub-task Resolved Lianet Magrans  
        102.
        Client support for java regex based subscription Sub-task In Progress Phuc Hong Tran  
        103.
        Group commit/callbacks triggering logic Sub-task Open Lianet Magrans  
        104.
        Review consumer leave group request best effort and response handling Sub-task Resolved Unassigned  
        105.
        Review minimal effort approach on consumer last heartbeat on unsubscribe Sub-task Open Lianet Magrans  
        106.
        Review consumer onPartitionsAssigned called with empty partitions Sub-task Open Lianet Magrans  
        107.
        Topic ID integration in consumer subscription state Sub-task Open Lianet Magrans  
        108.
        Update list group state type filter to include the states for the new consumer group type Sub-task Resolved Lan Ding  
        109.
        Handle timeouts for AsyncKafkaConsumer.commitSync Sub-task Resolved Unassigned  
        110.
        Review client logic for triggering offset commit callbacks Sub-task Open Lianet Magrans  
        111.
        group size counters do not reflect the actual sizes when operations fail Sub-task Open Jeff Kim  
        112.
        Ensure consumer does not start fetching from added partitions until onPartitionsAssigned completes Sub-task Resolved Lianet Magrans  
        113.
        Review client errors thrown on OffsetFetch and OffsetCommit failures Sub-task Resolved Lianet Magrans  
        114.
        Add Integration test for checking if the correct assignor is chosen Sub-task Open Unassigned  
        115.
        Coordinator unloading fails when replica is deleted Sub-task Resolved David Jacot  
        116.
        Partition is assigned to two members at the same time Sub-task Resolved David Jacot  
        117.
        Implement GroupMetadataManager#onUnloaded Sub-task Open Jeff Kim  
        118.
        System test failing for new consumer on endOffsets with negative timestamps Sub-task In Progress Philip Nee  
        119.
        Consumer invalid transition on expired poll interval Sub-task Resolved Lianet Magrans  
        120.
        Implement GroupCoordinator.onPartitionsDeleted Sub-task Resolved David Jacot  
        121.
        Fix client reconciliation of same assignment received in different epochs Sub-task Open Lianet Magrans  
        122.
        Extend admin to support ConsumerGroupDescribe API Sub-task Resolved David Jacot  
        123.
        Member should send full heartbeat when rejoining Sub-task Open Quoc Phong Dang  
        124.
        KafkaConsumer.groupMetadata() should be correct when first records are returned Sub-task Resolved Bruno Cadonna  
        125.
        Consumer does not rejoin after fenced on delayed revocation Sub-task Resolved Lianet Magrans  
        126.
        Fix handling of deleted topic when auto-committing before revocation Sub-task Open Lianet Magrans  
        127.
        Console consumer fails with `IllegalStateException` Sub-task Open Philip Nee  
        128.
        Move code style exceptions from suppressions.xml to the code Sub-task Open David Jacot  
        129.
        Improve reconciliation state machine Sub-task Open David Jacot  
        130.
        Fenced member should not send heartbeats while waiting for onPartitionsLost to complete Sub-task Patch Available Lianet Magrans  
        131.
        Stale member should trigger onPartitionsLost when leaving group Sub-task In Progress Lianet Magrans  

        Activity

          People

            dajac David Jacot
            dajac David Jacot
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 672h
                672h
                Remaining:
                Remaining Estimate - 672h
                672h
                Logged:
                Time Spent - Not Specified
                Not Specified