Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2651

Connection problem for KAFKA via ZOOKEEPER

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.9.4
    • Fix Version/s: 0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      cloudera 5.3

      Description

      When I try to get messages from kafka into hdfs file via FLUME, I have connections problems.

      metadata.broker.list must be used , props.put("zk.connect", "node2:2181") not supported.
      How can connect to kafka via zookeper.

      tier1.sources = source1
      tier1.channels = channel1
      tier1.sinks = sink1

      tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
      tier1.sources.source1.zookeeperConnect = quickstart.cloudera:2181
      tier1.sources.source1.topic = topic1
      tier1.sources.source1.groupId = flume
      tier1.sources.source1.channels = channel1
      tier1.sources.source1.interceptors = i1
      tier1.sources.source1.interceptors.i1.type = timestamp
      tier1.sources.source1.kafka.consumer.timeout.ms = 100

      tier1.channels.channel1.type = memory
      tier1.channels.channel1.capacity = 10000
      tier1.channels.channel1.transactionCapacity = 1000

      tier1.sinks.sink1.type = hdfs
      tier1.sinks.sink1.hdfs.path = /user/cloudera/kafka/%

      {topic}

      /%y-%m-%d
      tier1.sinks.sink1.hdfs.rollInterval = 5
      tier1.sinks.sink1.hdfs.rollSize = 0
      tier1.sinks.sink1.hdfs.rollCount = 0
      tier1.sinks.sink1.hdfs.fileType = DataStream
      tier1.sinks.sink1.channel = channel1


      actually, how can configure kafka listening port ?
      java

      Properties props = new Properties();
      ///props.put("zk.connect", "192.168.6.129:2181"); - not working reqires metadata.broker.list
      props.put("metadata.broker.list", "192.168.6.129:2181");

      props.put("serializer.class", "kafka.serializer.StringEncoder");

      ProducerConfig config = new ProducerConfig(props);
      producer = new Producer<String, String>(config);

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              smustaoglu Serdar Mustaoglu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: