[SAMZA-227] Upgrade KafkaSystemProducer to new API

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.9.0
• Component/s: kafka
• Labels: project
• Assignee: Navina Ramesh
• Reporter: Chris Riccomini
• Created: 06/Apr/14 00:27
• Updated: 09/Feb/15 04:10
• Resolved: 28/Jan/15 01:25

Description

Kafka has upgraded their producer API in 0.8.1/0.8.2 to have a new Java-based API. It's much improved both in performance, and features. We should upgrade KafkaSystemProducer to use it.

Initially, we should keep the old producer as well, and add a config to switch back and forth. Once we've run it for a while, and things look stable, we can deprecate and delete the old producer.

Issue Links

• is related to: KAFKA-1794 Make config and config defaults accessible to clients
• is related to: KAFKA-1863 Exception categories / hierarchy in clients
• is related to: SAMZA-459 Explicit flush for individual output streams
• relates to: SAMZA-523 Remove custom implementation of MockKafkaProducer, once the missing features are added to MockProducer in Kafka
• relates to: SAMZA-525 Update access to kafka's ProducerConfig variables
• relates to: KAFKA-1865 Add a flush() call to the new producer API
• relates to: SAMZA-522 Upgrade to Kafka 0.8.2 instead of Kafka 0.8.2-beta
• relates to: SAMZA-524 Update KafkaSystemProducer after Kafka provides a flush behavior
• relates to: SAMZA-546 Add config option max.in.flight.requests.per.connection