[GOBBLIN-190] Kafka Sink replication factor and partition creation.

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Component/s: gobblin-kafka
• Sprint: Apache Gobblin 170807, Apache Gobblin 170821, Apache Gobblin 170905
• Assignee: Vicky Kak
• Reporter: Vicky Kak
• Created: 06/Aug/17 04:54
• Updated: 03/Jan/18 06:52
• Resolved: 01/Nov/17 22:01

Description

The current implementation of the Kafka Writer does not allow the user to configure the replication-factor and partitions for the topic. The producer API creates a topic with the default configuration if the topic is not found when message is send to the topic. We may have to use the scala API's to configure the topics during creation for 0.8.1, for the 0.9.0 kafka there seems to be java apis.
The following link could be of help to this issue
https://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api