Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13681

Reduce Kafka dependencies in hadoop-kafka module

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: metrics
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Changed Apache Kafka dependency from kafka-2.10 to kafka-clients in hadoop-kafka module.

      Description

      The newly added Kafka module defines the Kafka dependency as:

      ...
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-2.10</artifactId>
      <version>${kafka.version}</version>
      ...
      

      This is unfavorable because its using the server dependency, which transitively has the client jars. The server dependency includes all of the server code and some larger transitive dependencies like Scala and Zookeeper.

      Instead the pom file should be changed to only depend on the clients jar which is a much smaller footprint:

      ...
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>${kafka.version}</version>
      ...
      

        Issue Links

          Activity

          Hide
          granthenke Grant Henke added a comment -

          I will provide a simple patch shortly

          Show
          granthenke Grant Henke added a comment - I will provide a simple patch shortly
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user granthenke opened a pull request:

          https://github.com/apache/hadoop/pull/133

          HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/granthenke/hadoop kafka-deps

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/133.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #133


          commit b2b738b362f7a8db1f80892e148ff6180bc901c4
          Author: Grant Henke <granthenke@gmail.com>
          Date: 2016-10-04T15:34:27Z

          HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user granthenke opened a pull request: https://github.com/apache/hadoop/pull/133 HADOOP-13681 . Reduce Kafka dependencies in hadoop-kafka module You can merge this pull request into a Git repository by running: $ git pull https://github.com/granthenke/hadoop kafka-deps Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #133 commit b2b738b362f7a8db1f80892e148ff6180bc901c4 Author: Grant Henke <granthenke@gmail.com> Date: 2016-10-04T15:34:27Z HADOOP-13681 . Reduce Kafka dependencies in hadoop-kafka module
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 0m 12s trunk passed
          +1 mvnsite 0m 15s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 12s the patch passed
          +1 compile 0m 11s the patch passed
          +1 javac 0m 11s the patch passed
          +1 mvnsite 0m 14s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 10s the patch passed
          +1 unit 0m 10s hadoop-kafka in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          10m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13681
          GITHUB PR https://github.com/apache/hadoop/pull/133
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 73719c6ce8f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef7f06f
          Default Java 1.8.0_101
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10657/testReport/
          modules C: hadoop-tools/hadoop-kafka U: hadoop-tools/hadoop-kafka
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10657/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 24s trunk passed +1 compile 0m 12s trunk passed +1 mvnsite 0m 15s trunk passed +1 mvneclipse 0m 12s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 12s the patch passed +1 compile 0m 11s the patch passed +1 javac 0m 11s the patch passed +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 10s the patch passed +1 unit 0m 10s hadoop-kafka in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 10m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13681 GITHUB PR https://github.com/apache/hadoop/pull/133 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 73719c6ce8f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef7f06f Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10657/testReport/ modules C: hadoop-tools/hadoop-kafka U: hadoop-tools/hadoop-kafka Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10657/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          granthenke Grant Henke added a comment -

          No added tests since this one line change is just reducing the dependencies.

          Show
          granthenke Grant Henke added a comment - No added tests since this one line change is just reducing the dependencies.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Nice catch! LGTM, +1.

          Show
          ajisakaa Akira Ajisaka added a comment - Nice catch! LGTM, +1.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/133

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/133
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk. Thanks Grant Henke for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk. Thanks Grant Henke for the contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10544/)
          HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. (aajisaka: rev 6cfe649375d2155cde2fbb49674ed2b4ba168a87)

          • (edit) hadoop-tools/hadoop-kafka/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10544/ ) HADOOP-13681 . Reduce Kafka dependencies in hadoop-kafka module. (aajisaka: rev 6cfe649375d2155cde2fbb49674ed2b4ba168a87) (edit) hadoop-tools/hadoop-kafka/pom.xml

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              granthenke Grant Henke
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development