Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2901

Document Kerberos setup for Kafka channel

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: Docs
    • Labels:
      None

      Description

      Add details about configuring Kafka channel to work with a Kerberized Kafka cluster

      1. FLUME-2901.patch
        3 kB
        Roshan Naik
      2. FLUME-2901.v2.patch
        10 kB
        Roshan Naik
      3. FLUME-2901.v3.patch
        10 kB
        Roshan Naik

        Activity

        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-1 #193 (See https://builds.apache.org/job/Flume-trunk-hbase-1/193/)
        FLUME-2901. Document Kerberos setup for Kafka channel (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c1fae53bf2c254f5cf1282f32116a7aedc5340ed)

        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-1 #193 (See https://builds.apache.org/job/Flume-trunk-hbase-1/193/ ) FLUME-2901 . Document Kerberos setup for Kafka channel (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c1fae53bf2c254f5cf1282f32116a7aedc5340ed ) flume-ng-doc/sphinx/FlumeUserGuide.rst
        Hide
        mpercy Mike Percy added a comment -

        Also thank you Jeff for reviewing.

        Show
        mpercy Mike Percy added a comment - Also thank you Jeff for reviewing.
        Hide
        mpercy Mike Percy added a comment -

        Pushed to trunk. Thanks for the patch Roshan!

        Show
        mpercy Mike Percy added a comment - Pushed to trunk. Thanks for the patch Roshan!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c1fae53bf2c254f5cf1282f32116a7aedc5340ed in flume's branch refs/heads/trunk from Roshan Naik
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c1fae53 ]

        FLUME-2901. Document Kerberos setup for Kafka channel

        Add details about configuring Kafka channel to work with a Kerberized
        Kafka cluster.

        (Roshan Naik via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit c1fae53bf2c254f5cf1282f32116a7aedc5340ed in flume's branch refs/heads/trunk from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c1fae53 ] FLUME-2901 . Document Kerberos setup for Kafka channel Add details about configuring Kafka channel to work with a Kerberized Kafka cluster. (Roshan Naik via Mike Percy)
        Hide
        mpercy Mike Percy added a comment -

        +1. I am going to commit this.

        Show
        mpercy Mike Percy added a comment - +1. I am going to commit this.
        Hide
        jholoman Jeff Holoman added a comment -

        LGTM +1

        Show
        jholoman Jeff Holoman added a comment - LGTM +1
        Hide
        roshan_naik Roshan Naik added a comment -

        uploading revised patch v3 with fix for issue pointed out by Jeff Holoman

        Show
        roshan_naik Roshan Naik added a comment - uploading revised patch v3 with fix for issue pointed out by Jeff Holoman
        Hide
        roshan_naik Roshan Naik added a comment -

        Jeff Holoman thanks for spotting that problem .. shall upload a fixed patch soon.

        I considered providing examples for other modes too but was not able to test them out. After briefly looking over the other modes, leaned away from providing examples for each of them and instead just refer to them via a link... as it seemed a bit much (e.g. with and without authorization) although useful.

        I decided to put this example since the kafka folks tell me that SASL_PLAINTEXT is most common one.

        Show
        roshan_naik Roshan Naik added a comment - Jeff Holoman thanks for spotting that problem .. shall upload a fixed patch soon. I considered providing examples for other modes too but was not able to test them out. After briefly looking over the other modes, leaned away from providing examples for each of them and instead just refer to them via a link... as it seemed a bit much (e.g. with and without authorization) although useful. I decided to put this example since the kafka folks tell me that SASL_PLAINTEXT is most common one.
        Hide
        jholoman Jeff Holoman added a comment -

        Looks ok but the following lines need to be addressed:

        a1.channels.c1.kafka.producer.security.protocol = SASL_PLAINTEXT
        + a1.channels.c1.kafka.consumer.security.protocol = SASL_PLAINTEXT

        s/c1/channel1 per the previous lines.

        Also if we are going to take the time to document this does it make sense to include an example configuration with SASL_SSL as well? I see that it was called out in the change(i.e, look at the kafka docs) and that's probably fine. I could go either way really.

        Show
        jholoman Jeff Holoman added a comment - Looks ok but the following lines need to be addressed: a1.channels.c1.kafka.producer.security.protocol = SASL_PLAINTEXT + a1.channels.c1.kafka.consumer.security.protocol = SASL_PLAINTEXT s/c1/channel1 per the previous lines. Also if we are going to take the time to document this does it make sense to include an example configuration with SASL_SSL as well? I see that it was called out in the change(i.e, look at the kafka docs) and that's probably fine. I could go either way really.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        I am not too familiar with the new kafka API. Jeff Holoman - could you take a look?

        Show
        hshreedharan Hari Shreedharan added a comment - I am not too familiar with the new kafka API. Jeff Holoman - could you take a look?
        Hide
        roshan_naik Roshan Naik added a comment -

        Hari Shreedharan can u take a look ?

        Show
        roshan_naik Roshan Naik added a comment - Hari Shreedharan can u take a look ?
        Hide
        roshan_naik Roshan Naik added a comment -

        revising patch with minor addition to documentation

        Show
        roshan_naik Roshan Naik added a comment - revising patch with minor addition to documentation
        Hide
        roshan_naik Roshan Naik added a comment -

        Uploading patch. includes Kafka channel kerberos documentation and also a minor fix to existing doc (removed capacity & transactionCapacity settings from example as they dont apply to this channel)

        Show
        roshan_naik Roshan Naik added a comment - Uploading patch. includes Kafka channel kerberos documentation and also a minor fix to existing doc (removed capacity & transactionCapacity settings from example as they dont apply to this channel)

          People

          • Assignee:
            roshan_naik Roshan Naik
            Reporter:
            roshan_naik Roshan Naik
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development