Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3323

Kafka channel should supports disable producer/consumer according to the need

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.9.0
    • Fix Version/s: None
    • Component/s: Kafka Channel
    • Labels:
      None

      Description

      As the doc said:

      The Kafka channel can be used for multiple scenarios:
      
      1. With Flume source and sink - it provides a reliable and highly available channel for events
      2. With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps
      3. With Flume sink, but no source - it is a low-latency, fault tolerant way to send events from Kafka to Flume sinks such as HDFS, HBase or Solr

       

      Kafka channel will generates producer and consumer both now.

      When using Kafka channel as scenario 2 (3), the producer(consumer) is just useless, and if our kafka cluster enable SASL the situation is getting worse.

      Assume we have a kafka account which only has read permissions, and using kafka channel without source, producer shouldn't be generated.

      I propose that add two properties to disable producer/consumer.

       

      agent.channels.c.noProducer = false
      agent.channels.c.noConsumer= false

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhipcui zhipcui
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: