Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2500

Add a channel that uses Kafka

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • None
    • None

    Description

      Here is the rationale:

      • Kafka does give a HA channel, which means a dead agent does not affect the data in the channel - thus reducing delay of delivery.
      • Kafka is used by many companies - it would be a good idea to use Flume to pull data from Kafka and write it to HDFS/HBase etc.

      This channel is not going to be useful for cases where Kafka is not already used, since it brings is operational overhead of maintaining two systems, but if there is Kafka in use - this is good way to integrate Kafka and Flume.

      Here is an a scratch implementation: https://github.com/harishreedharan/flume/blob/kafka-channel/flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java

      Attachments

        1. FLUME-2500.patch
          48 kB
          Hari Shreedharan
        2. FLUME-2500.patch
          49 kB
          Hari Shreedharan
        3. FLUME-2500-1.patch
          49 kB
          Hari Shreedharan
        4. FLUME-2500-2.patch
          48 kB
          Hari Shreedharan

        Issue Links

          Activity

            People

              hshreedharan Hari Shreedharan
              hshreedharan Hari Shreedharan
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: