Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13681

Reduce Kafka dependencies in hadoop-kafka module

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: metrics
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Changed Apache Kafka dependency from kafka-2.10 to kafka-clients in hadoop-kafka module.

      Description

      The newly added Kafka module defines the Kafka dependency as:

      ...
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-2.10</artifactId>
      <version>${kafka.version}</version>
      ...
      

      This is unfavorable because its using the server dependency, which transitively has the client jars. The server dependency includes all of the server code and some larger transitive dependencies like Scala and Zookeeper.

      Instead the pom file should be changed to only depend on the clients jar which is a much smaller footprint:

      ...
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>${kafka.version}</version>
      ...
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                granthenke Grant Henke
                Reporter:
                granthenke Grant Henke
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: