[FLUME-2651] Connection problem for KAFKA via ZOOKEEPER

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources

Environment

cloudera 5.3

• Assignee: Unassigned
• Reporter: Serdar Mustaoglu
• Created: 25/Mar/15 11:34
• Updated: 21/Jun/16 20:47
• Resolved: 21/Jun/16 20:47

Description

When I try to get messages from kafka into hdfs file via FLUME, I have connections problems.

metadata.broker.list must be used , props.put("zk.connect", "node2:2181") not supported.
How can connect to kafka via zookeper.

tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1

tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
tier1.sources.source1.zookeeperConnect = quickstart.cloudera:2181
tier1.sources.source1.topic = topic1
tier1.sources.source1.groupId = flume
tier1.sources.source1.channels = channel1
tier1.sources.source1.interceptors = i1
tier1.sources.source1.interceptors.i1.type = timestamp
tier1.sources.source1.kafka.consumer.timeout.ms = 100

tier1.channels.channel1.type = memory
tier1.channels.channel1.capacity = 10000
tier1.channels.channel1.transactionCapacity = 1000

tier1.sinks.sink1.type = hdfs
tier1.sinks.sink1.hdfs.path = /user/cloudera/kafka/%

{topic}

/%y-%m-%d
tier1.sinks.sink1.hdfs.rollInterval = 5
tier1.sinks.sink1.hdfs.rollSize = 0
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.channel = channel1

---

actually, how can configure kafka listening port ?
java

Properties props = new Properties();
///props.put("zk.connect", "192.168.6.129:2181"); - not working reqires metadata.broker.list
props.put("metadata.broker.list", "192.168.6.129:2181");

props.put("serializer.class", "kafka.serializer.StringEncoder");

ProducerConfig config = new ProducerConfig(props);
producer = new Producer<String, String>(config);