[GEARPUMP-40] KafkaSource poor performance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Component/s: kafka
• Assignee: Manu Zhang
• Reporter: Manu Zhang
• Created: 20/Apr/16 01:38
• Updated: 03/Jun/16 07:33

Description

As per https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines, a single thread Kafka consumer could consume at 89.7MB/s from 6x partition 3x replica topic whose data are evenly distributed on a 3-node GbE cluster. I carried out a similar experiment with KafkaSource and found that the throughput is only at 10MB/s.