Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5649

ExecuteFlumeSource kafka source cant connect to topics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.1
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      Debian 8

      Description

      I use Nifi's webui to create an ExecuteFlumeSource Processor. The configuration is following:

      Source Type: org.apache.flume.source.kafka.KafkaSource
      Agent Name: l12agent1
      Source Name: source1
      Flume Configuration: 
      l12agent1.sources.source1.kafka.bootstrap.servers = 10.90.1.111:9092
      l12agent1.sources.source1.kafka.topics = l12_source
      l12agent1.sources.source1.kafka.consumer.group.id = flume_nifi_l12
      l12agent1.sources.source1.kafka.consumer.auto.offset.reset = latest
      l12agent1.sources.source1.batchSize = 10000
      l12agent1.sources.source1.batchDurationMillis = 5000
      

       However nifi gives error that " kafka topic needs to be specified" but I do have that topic created and the flume agent run well from command line.

      2018-09-30 11:12:16,503 ERROR [Timer-Driven Process Thread-1] o.a.n.p.flume.ExecuteFlumeSource ExecuteFlumeSource[id=2854
      1a45-0166-1000-ccda-fb75266203b5] Error creating source: org.apache.flume.conf.ConfigurationException: Kafka topic must b
      e specified. 
      org.apache.flume.conf.ConfigurationException: Kafka topic must be specified. 
             at org.apache.flume.source.kafka.KafkaSource.configure(KafkaSource.java:180) 
             at org.apache.flume.conf.Configurables.configure(Configurables.java:41) 
             at org.apache.nifi.processors.flume.ExecuteFlumeSource.onScheduled(ExecuteFlumeSource.java:137) 
             at sun.reflect.GeneratedMethodAccessor376.invoke(Unknown Source) 
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
             at java.lang.reflect.Method.invoke(Method.java:498) 
             at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142) 
             at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130) 
             at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75) 
             at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52) 
             at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499) 
             at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
             at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.ja
      va:180) 
             at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
             at java.lang.Thread.run(Thread.java:748)
      
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ysn2233 Shengnan YU
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: