Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2470

Kafka Sink and Source must use camel case for all configs.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Some configs are of the form kafka.* while others are camel cased. This is for ease of implementation as these get passed to Kafka, but this make config params not uniform. We must change this so all configs are camel-cased. We should translate these camel-cased params to the kafka properties to configure the kafka API.

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #661 (See https://builds.apache.org/job/flume-trunk/661/)
        FLUME-2470. Kafka Sink and Source must use camel case for all configs. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bde2c28211a2d05a9930f1599cb15864ad3cdba0)

        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/kafka-server.properties
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceUtilTest.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/log4j.properties
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java
          FLUME-2470. Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=26444fd7a8c804b6f2507ee3bf54e1c811ee5168)
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/KafkaSinkUtilTest.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkUtil.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #661 (See https://builds.apache.org/job/flume-trunk/661/ ) FLUME-2470 . Kafka Sink and Source must use camel case for all configs. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bde2c28211a2d05a9930f1599cb15864ad3cdba0 ) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/kafka-server.properties flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceUtilTest.java flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/log4j.properties flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java FLUME-2470 . Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=26444fd7a8c804b6f2507ee3bf54e1c811ee5168 ) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/KafkaSinkUtilTest.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkUtil.java
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-98 #21 (See https://builds.apache.org/job/Flume-trunk-hbase-98/21/)
        FLUME-2470. Kafka Sink and Source must use camel case for all configs. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bde2c28211a2d05a9930f1599cb15864ad3cdba0)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/kafka-server.properties
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/log4j.properties
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceUtilTest.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
          FLUME-2470. Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=26444fd7a8c804b6f2507ee3bf54e1c811ee5168)
        • flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkUtil.java
        • flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/KafkaSinkUtilTest.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #21 (See https://builds.apache.org/job/Flume-trunk-hbase-98/21/ ) FLUME-2470 . Kafka Sink and Source must use camel case for all configs. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=bde2c28211a2d05a9930f1599cb15864ad3cdba0 ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/kafka-server.properties flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java flume-ng-sinks/flume-ng-kafka-sink/src/test/resources/log4j.properties flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceUtilTest.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java FLUME-2470 . Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=26444fd7a8c804b6f2507ee3bf54e1c811ee5168 ) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkUtil.java flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/KafkaSinkUtilTest.java
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 94d8ea40a6e08b53ae04b3f056944b5a99cf66d2 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=94d8ea4 ]

        FLUME-2470. Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them.

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 94d8ea40a6e08b53ae04b3f056944b5a99cf66d2 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=94d8ea4 ] FLUME-2470 . Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (Gwen Shapira via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 26444fd7a8c804b6f2507ee3bf54e1c811ee5168 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=26444fd ]

        FLUME-2470. Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them.

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 26444fd7a8c804b6f2507ee3bf54e1c811ee5168 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=26444fd ] FLUME-2470 . Kafka Sink and Source config updates. Missed a couple files in the last commit, adding them. (Gwen Shapira via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Gwen!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Gwen!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 56c1b866213aeca3f57d7bbd46d9214e082182ae in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=56c1b86 ]

        FLUME-2470. Kafka Sink and Source must use camel case for all configs.

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 56c1b866213aeca3f57d7bbd46d9214e082182ae in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=56c1b86 ] FLUME-2470 . Kafka Sink and Source must use camel case for all configs. (Gwen Shapira via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit bde2c28211a2d05a9930f1599cb15864ad3cdba0 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bde2c28 ]

        FLUME-2470. Kafka Sink and Source must use camel case for all configs.

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit bde2c28211a2d05a9930f1599cb15864ad3cdba0 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bde2c28 ] FLUME-2470 . Kafka Sink and Source must use camel case for all configs. (Gwen Shapira via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Committing this.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Committing this.
        Hide
        gwenshap Gwen Shapira added a comment -

        It is. Since the new parameters change documentation, it made sense to assume the docs are already there.

        Show
        gwenshap Gwen Shapira added a comment - It is. Since the new parameters change documentation, it made sense to assume the docs are already there.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Is this dependent on FLUME-2455?

        Show
        hshreedharan Hari Shreedharan added a comment - Is this dependent on FLUME-2455 ?
        Hide
        hshreedharan Hari Shreedharan added a comment -

        The patch does not apply. Could you please rebase?

        Show
        hshreedharan Hari Shreedharan added a comment - The patch does not apply. Could you please rebase?
        Hide
        gwenshap Gwen Shapira added a comment -

        Patch standardizing the parameters.

        Mandatory parameters for Kafka source/sink are now camelCase.
        Any Kafka parameter can still be passed along using the "kafka."+kafka.parameter.name convention.

        I modified the documentation to match, cleaned up the configuration code a bit to support the new logic and added a test to validate.

        Show
        gwenshap Gwen Shapira added a comment - Patch standardizing the parameters. Mandatory parameters for Kafka source/sink are now camelCase. Any Kafka parameter can still be passed along using the "kafka."+kafka.parameter.name convention. I modified the documentation to match, cleaned up the configuration code a bit to support the new logic and added a test to validate.

          People

          • Assignee:
            gwenshap Gwen Shapira
            Reporter:
            hshreedharan Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development