[HADOOP-10115] Exclude duplicate jars in hadoop package under different component's lib

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha1
• Component/s: build
• Labels: common hdfs mapreduce nfs yarn
• Hadoop Flags: Incompatible change
• Release Note: Jars in the various subproject lib directories are now de-duplicated against Hadoop common. Users who interact directly with those directories must be sure to pull in common's dependencies as well.
• Assignee: Vinayakumar B
• Reporter: Vinayakumar B
• Created: 19/Nov/13 06:20
• Updated: 12/May/16 18:22
• Resolved: 10/Mar/15 04:44

Description

In the hadoop package distribution there are more than 90% of the jars are duplicated in multiple places.
For Ex:
almost all jars in share/hadoop/hdfs/lib are already there in share/hadoop/common/lib

Same case for all other lib in share directory.

Anyway for all the daemon processes all directories are added to classpath.

So to reduce the package distribution size and the classpath overhead, remove the duplicate jars from the distribution.

Issue Links

• duplicates: HADOOP-12363 Hadoop binary distributions contain many copies of the same jars
• duplicates: HADOOP-11680 Deduplicate jars in convenience binary distribution