Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1326

New consumer checklist

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.1
    • Fix Version/s: 0.9.0.1
    • Component/s: consumer
    • Labels:

      Description

      We will use this JIRA to track the list of issues to resolve to get a working new consumer client. The consumer client can work in phases -

      1. Add new consumer APIs and configs
      2. Refactor Sender. We will need to use some common APIs from Sender.java (https://issues.apache.org/jira/browse/KAFKA-1316)
      3. Add metadata fetch and refresh functionality to the consumer (This will require https://issues.apache.org/jira/browse/KAFKA-1316)
      4. Add functionality to support subscribe(TopicPartition...partitions). This will add SimpleConsumer functionality to the new consumer. This does not include any group management related work.
      5. Add ability to commit offsets to Kafka. This will include adding functionality to the commit()/commitAsync()/committed() APIs. This still does not include any group management related work.
      6. Add functionality to the offsetsBeforeTime() API.
      7. Add consumer co-ordinator election to the server. This will only add a new module for the consumer co-ordinator, but not necessarily all the logic to do group management.

      At this point, we will have a fully functional standalone consumer and a server side co-ordinator module. This will be a good time to start adding group management functionality to the server and consumer.

      8. Add failure detection capability to the consumer when group management is used. This will not include any rebalancing logic, just the ability to detect failures using session.timeout.ms.
      9. Add rebalancing logic to the server and consumer. This will be a tricky and potentially large change since it will involve implementing the group management protocol.
      10. Add system tests for the new consumer
      11. Add metrics
      12. Convert mirror maker to use the new consumer.
      13. Convert perf test to use the new consumer
      14. Performance testing and analysis.
      15. Review and fine tune log4j logging

        Attachments

          Issue Links

          1.
          Add new consumer APIs Sub-task Resolved Neha Narkhede
          2.
          Refactor Sender Sub-task Resolved Jay Kreps
          3.
          Add new request and response formats for the new consumer and coordinator communication Sub-task Resolved Jun Rao
          4.
          Add metadata fetch and refresh functionality to the consumer Sub-task Resolved Neha Narkhede
          5.
          Implement subscribe(TopicPartition...partitions) and pool(timeout) in the consumer Sub-task Resolved Jay Kreps
          6.
          Add ability to commit offsets to the new consumer Sub-task Resolved Jay Kreps
          7.
          Add consumer co-ordinator module to the server Sub-task Resolved Guozhang Wang
          8.
          Coordinator should detect consumer failures Sub-task Resolved Onur Karaman
          9.
          Add rebalancing logic to the coordinator / consumer Sub-task Resolved Guozhang Wang
          10.
          Implement new consumer client Sub-task Resolved Jay Kreps
          11.
          Refactor KafkaConsumer Sub-task Resolved Guozhang Wang
          12.
          Coordinator Node Id set to INT_MAX breaks coordinator metadata updates Sub-task Resolved Guozhang Wang
          13.
          Consumer should detect coordinator failures Sub-task Resolved Guozhang Wang
          14.
          Merge Offset manager into Coordinator Sub-task Resolved Guozhang Wang
          15.
          Allow regex subscriptions in the new consumer Sub-task Resolved Ashish Singh
          16.
          Enable ConsoleConsumer to use new consumer Sub-task Resolved Ben Stopford
          17.
          New consumer poll() can block other calls like position(), commit(), and close() indefinitely Sub-task Resolved Jason Gustafson
          18.
          Add a ListTopics() API to the new consumer Sub-task Resolved Ashish Singh
          19.
          Make new consumer offset commit API use callback + future Sub-task Resolved Jason Gustafson
          20.
          Add integration test for consumer coordinator Sub-task Resolved Jason Gustafson
          21.
          Add response tests for ConsumerCoordinator Sub-task Resolved Onur Karaman
          22.
          Add functionality to the offsetsBeforeTime() API Sub-task Resolved Jason Gustafson
          23.
          System tests for the new consumer and co-ordinator Sub-task Resolved Guozhang Wang
          24.
          Persist Coordinator State for Coordinator Failover Sub-task Resolved Guozhang Wang
          25.
          Add metadata to consumer registry info Sub-task Resolved Guozhang Wang
          26.
          Allow coordinator to aggresively remove group when it is no longer owning the group Sub-task Resolved Guozhang Wang
          27.
          KIP-54: Add rebalance with a minimal number of reassignments to server-defined strategy list Sub-task Resolved Vahid Hashemian
          28.
          leave group request Sub-task Resolved Onur Karaman
          29.
          enable new consumer in mirror maker Sub-task Resolved Jiangjie Qin
          30.
          enable new consumer in EndToEndLatency Sub-task Resolved Ben Stopford
          31.
          Client-side assignment and group generalization Sub-task Resolved Jason Gustafson
          32.
          New consumer performance Sub-task Resolved Jason Gustafson
          33.
          change kafka.examples.Consumer to use the new java consumer Sub-task Resolved Ashish Singh
          34.
          support new consumer in ConsumerGroupCommand Sub-task Resolved Ashish Singh
          35.
          Update ErrorMapping with New Consumer Errors Sub-task Resolved Jason Gustafson
          36.
          handle oversized messages properly in new consumer Sub-task Resolved Guozhang Wang
          37.
          Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll` Sub-task Resolved Ismael Juma
          38.
          Move fetchable check from fetchedRecords to fetch response handler Sub-task Resolved Guozhang Wang
          39.
          Consumer.poll(0)'s overhead too large Sub-task Resolved Jason Gustafson
          40.
          add leave group logic to the consumer Sub-task Resolved Jason Gustafson
          41.
          add paused API Sub-task Resolved Jason Gustafson
          42.
          Periodic purging groups in the coordinator Sub-task Resolved Jason Gustafson
          43.
          Update JavaDoc for the new / updated APIs Sub-task Resolved Guozhang Wang
          44.
          Add an API to new consumer to allow user get high watermark of partitions. Sub-task Resolved Jiangjie Qin

            Activity

              People

              • Assignee:
                nehanarkhede Neha Narkhede
                Reporter:
                nehanarkhede Neha Narkhede
              • Votes:
                3 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: