[KAFKA-1326] New consumer checklist

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.2.1
• Fix Version/s: 0.9.0.1
• Component/s: consumer
• Labels: feature
• Assignee: Neha Narkhede
• Reporter: Neha Narkhede
• Created: 25/Mar/14 19:54
• Updated: 06/Nov/17 17:47
• Resolved: 06/Nov/17 17:47

Description

We will use this JIRA to track the list of issues to resolve to get a working new consumer client. The consumer client can work in phases -

1. Add new consumer APIs and configs
2. Refactor Sender. We will need to use some common APIs from Sender.java (https://issues.apache.org/jira/browse/KAFKA-1316)
3. Add metadata fetch and refresh functionality to the consumer (This will require https://issues.apache.org/jira/browse/KAFKA-1316)
4. Add functionality to support subscribe(TopicPartition...partitions). This will add SimpleConsumer functionality to the new consumer. This does not include any group management related work.
5. Add ability to commit offsets to Kafka. This will include adding functionality to the commit()/commitAsync()/committed() APIs. This still does not include any group management related work.
6. Add functionality to the offsetsBeforeTime() API.
7. Add consumer co-ordinator election to the server. This will only add a new module for the consumer co-ordinator, but not necessarily all the logic to do group management.

At this point, we will have a fully functional standalone consumer and a server side co-ordinator module. This will be a good time to start adding group management functionality to the server and consumer.

8. Add failure detection capability to the consumer when group management is used. This will not include any rebalancing logic, just the ability to detect failures using session.timeout.ms.
9. Add rebalancing logic to the server and consumer. This will be a tricky and potentially large change since it will involve implementing the group management protocol.
10. Add system tests for the new consumer
11. Add metrics
12. Convert mirror maker to use the new consumer.
13. Convert perf test to use the new consumer
14. Performance testing and analysis.
15. Review and fine tune log4j logging

Issue Links

• duplicates: KAFKA-360 Add ability to disable rebalancing in ZooKeeper consumer
• duplicates: KAFKA-361 Add ability to get offsets from ZK-based consumer
• duplicates: KAFKA-389 Add a consumer client that connects to the co-ordinator for rebalancing
• duplicates: KAFKA-390 Implement the rebalancing protocol between co-ordinator and consumer
• duplicates: KAFKA-364 Consumer re-design
• duplicates: KAFKA-387 Add a centralized co-ordinator for consumer rebalancing
• incorporates: KAFKA-1655 Allow high performance SimpleConsumer use cases to still work with new Kafka 0.9 consumer APIs
• is duplicated by: KAFKA-167 Move partition assignment to the broker
• requires: KAFKA-2387 Improve KafkaConsumer API

Sub-Tasks

• Add new consumer APIs: Resolved
• Refactor Sender: Resolved
• Add new request and response formats for the new consumer and coordinator communication: Resolved
• Add metadata fetch and refresh functionality to the consumer: Resolved
• Implement subscribe(TopicPartition...partitions) and pool(timeout) in the consumer: Resolved
• Add ability to commit offsets to the new consumer: Resolved
• Add consumer co-ordinator module to the server: Resolved
• Coordinator should detect consumer failures: Resolved
• Add rebalancing logic to the coordinator / consumer: Resolved
• Implement new consumer client: Resolved
• Refactor KafkaConsumer: Resolved
• Coordinator Node Id set to INT_MAX breaks coordinator metadata updates: Resolved
• Consumer should detect coordinator failures: Resolved
• Merge Offset manager into Coordinator: Resolved
• Allow regex subscriptions in the new consumer: Resolved
• Enable ConsoleConsumer to use new consumer: Resolved
• New consumer poll() can block other calls like position(), commit(), and close() indefinitely: Resolved
• Add a ListTopics() API to the new consumer: Resolved
• Make new consumer offset commit API use callback + future: Resolved
• Add integration test for consumer coordinator: Resolved
• Add response tests for ConsumerCoordinator: Resolved
• Add functionality to the offsetsBeforeTime() API: Resolved
• System tests for the new consumer and co-ordinator: Resolved
• Persist Coordinator State for Coordinator Failover: Resolved
• Add metadata to consumer registry info: Resolved
• Allow coordinator to aggresively remove group when it is no longer owning the group: Resolved
• KIP-54: Add rebalance with a minimal number of reassignments to server-defined strategy list: Resolved
• leave group request: Resolved
• enable new consumer in mirror maker: Resolved
• enable new consumer in EndToEndLatency: Resolved
• Client-side assignment and group generalization: Resolved
• New consumer performance: Resolved
• change kafka.examples.Consumer to use the new java consumer: Resolved
• support new consumer in ConsumerGroupCommand: Resolved
• Update ErrorMapping with New Consumer Errors: Resolved
• handle oversized messages properly in new consumer: Resolved
• Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll`: Resolved
• Move fetchable check from fetchedRecords to fetch response handler: Resolved
• Consumer.poll(0)'s overhead too large: Resolved
• add leave group logic to the consumer: Resolved
• add paused API: Resolved
• Periodic purging groups in the coordinator: Resolved
• Update JavaDoc for the new / updated APIs: Resolved
• Add an API to new consumer to allow user get high watermark of partitions.: Resolved