[HADOOP-13681] Reduce Kafka dependencies in hadoop-kafka module

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha2
• Component/s: metrics
• Hadoop Flags: Incompatible change
• Release Note: Changed Apache Kafka dependency from kafka-2.10 to kafka-clients in hadoop-kafka module.
• Assignee: Grant Henke
• Reporter: Grant Henke
• Created: 04/Oct/16 15:30
• Updated: 26/Jun/19 01:03
• Resolved: 05/Oct/16 11:47

Description

The newly added Kafka module defines the Kafka dependency as:

...
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-2.10</artifactId>
<version>${kafka.version}</version>
...

This is unfavorable because its using the server dependency, which transitively has the client jars. The server dependency includes all of the server code and some larger transitive dependencies like Scala and Zookeeper.

Instead the pom file should be changed to only depend on the clients jar which is a much smaller footprint:

...
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
...

Issue Links

• relates to: HADOOP-10949 metrics2 sink plugin for Apache Kafka
• links to: GitHub Pull Request #133
• links to: GitHub Pull Request #133