Kafka 0.8.2 supports native offsets management instead of ZK, this will get better performance, for now in ReliableKafkaReceiver, we rely on ZK to manage the offsets, this potentially will be a bottleneck if the injection rate is high (once per 200ms by default), so here in order to get better performance as well as keeping consistent with Kafka, add native offset management for ReliableKafkaReceiver.
- depends upon
-
SPARK-2808 update kafka to version 0.8.2
-
- Resolved
-