Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Docs
    • Labels:
      None

      Description

      FLUME-2500 adds a Kafka channel. We need to document its usage.

      1. FLUME-2523.0.patch
        4 kB
        Gwen Shapira
      2. FLUME-2523.1.patch
        5 kB
        Gwen Shapira

        Activity

        Hide
        gwenshap Gwen Shapira added a comment -

        Documentation for Kafka Channel

        Show
        gwenshap Gwen Shapira added a comment - Documentation for Kafka Channel
        Hide
        paliwalashish Ashish Paliwal added a comment -

        +1

        Minor one, if we can put the URI format for zookeeper as well

        *zookeeperConnect* – URI of ZooKeeper used by Kafka cluster. The format is comma separated list of hostname:port

        Show
        paliwalashish Ashish Paliwal added a comment - +1 Minor one, if we can put the URI format for zookeeper as well * zookeeperConnect * – URI of ZooKeeper used by Kafka cluster. The format is comma separated list of hostname:port
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Thanks for writing these docs!

        A few minor comments:

        • The example shows kafka-1:9092 etc. That should be a full hostname, correct? (for both ZK and broker list).
        • Mention that class that is used when parseAsFlumeEvent is true. This would be required if we just want Flume to write to Kafka and not remove it from the channel - so other consumers are able to remove the data (Log4j->Flume->Kafka etc). The class is org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact.
        • Worth mentioning how multiple channels must use the same topic and group to ensure that when one agent fails another can remove the data.
        • readSmallestOffset matters only if parseAsFlumeEvent is false. We should also mention why it is better to keep it to false (or not set it at all) if parseAsFlumeEvent is false.
        • Mention that producer.type will always be sync and other properties which we override anyway - even if the customer has overriden it.
        • Mention that by using Kafka Channel -> HDFS/HBase/Solr sink, Kafka+Flume provides low latency, fault-tolerant writes to the corresponding system.
        Show
        hshreedharan Hari Shreedharan added a comment - Thanks for writing these docs! A few minor comments: The example shows kafka-1:9092 etc. That should be a full hostname, correct? (for both ZK and broker list). Mention that class that is used when parseAsFlumeEvent is true. This would be required if we just want Flume to write to Kafka and not remove it from the channel - so other consumers are able to remove the data (Log4j->Flume->Kafka etc). The class is org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact. Worth mentioning how multiple channels must use the same topic and group to ensure that when one agent fails another can remove the data. readSmallestOffset matters only if parseAsFlumeEvent is false. We should also mention why it is better to keep it to false (or not set it at all) if parseAsFlumeEvent is false. Mention that producer.type will always be sync and other properties which we override anyway - even if the customer has overriden it. Mention that by using Kafka Channel -> HDFS/HBase/Solr sink, Kafka+Flume provides low latency, fault-tolerant writes to the corresponding system.
        Hide
        gwenshap Gwen Shapira added a comment -

        Thanks for the detailed review!

        Fixed most issues raised by Ashish and Hari.

        Exceptions:
        1) any hostname that resolves is valid for zookeeper and broker. I didn't include FQD for brevity and clarity
        2) I'd rather not document properties that we don't want users to override.
        Lets view it as internal implementation detail? I feel that documenting invites fiddling, and I'd rather they won't fiddle here.

        Show
        gwenshap Gwen Shapira added a comment - Thanks for the detailed review! Fixed most issues raised by Ashish and Hari. Exceptions: 1) any hostname that resolves is valid for zookeeper and broker. I didn't include FQD for brevity and clarity 2) I'd rather not document properties that we don't want users to override. Lets view it as internal implementation detail? I feel that documenting invites fiddling, and I'd rather they won't fiddle here.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. There were a couple of formatting errors. I fixed those and am committing it.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. There were a couple of formatting errors. I fixed those and am committing it.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit fdd61e840b004ca1b5c23b28e8d9eeda300f56f7 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fdd61e8 ]

        FLUME-2523. Document Kafka channel

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit fdd61e840b004ca1b5c23b28e8d9eeda300f56f7 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fdd61e8 ] FLUME-2523 . Document Kafka channel (Gwen Shapira via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ae2ded9ce6e112f8a423853e6781b1e02862ab3c in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ae2ded9 ]

        FLUME-2523. Document Kafka channel

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit ae2ded9ce6e112f8a423853e6781b1e02862ab3c in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ae2ded9 ] FLUME-2523 . Document Kafka channel (Gwen Shapira via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Gwen!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Gwen!
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-98 #43 (See https://builds.apache.org/job/Flume-trunk-hbase-98/43/)
        FLUME-2523. Document Kafka channel (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fdd61e840b004ca1b5c23b28e8d9eeda300f56f7)

        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #43 (See https://builds.apache.org/job/Flume-trunk-hbase-98/43/ ) FLUME-2523 . Document Kafka channel (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fdd61e840b004ca1b5c23b28e8d9eeda300f56f7 ) flume-ng-doc/sphinx/FlumeUserGuide.rst
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #686 (See https://builds.apache.org/job/flume-trunk/686/)
        FLUME-2523. Document Kafka channel (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fdd61e840b004ca1b5c23b28e8d9eeda300f56f7)

        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #686 (See https://builds.apache.org/job/flume-trunk/686/ ) FLUME-2523 . Document Kafka channel (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fdd61e840b004ca1b5c23b28e8d9eeda300f56f7 ) flume-ng-doc/sphinx/FlumeUserGuide.rst

          People

          • Assignee:
            gwenshap Gwen Shapira
            Reporter:
            gwenshap Gwen Shapira
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development