[HADOOP-12857] Rework hadoop-tools

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha1
• Component/s: build
• Hadoop Flags: Incompatible change

Release Note

<!-- markdown -->
* Turning on optional things from the tools directory such as S3 support can now be done in hadoop-env.sh with the HADOOP\_OPTIONAL\_TOOLS environment variable without impacting the various user-facing CLASSPATH variables.
* The tools directory is no longer pulled in blindly for any utilities that pull it in.
* TOOL\_PATH / HADOOP\_TOOLS\_PATH has been broken apart and replaced with HADOOP\_TOOLS\_HOME, HADOOP\_TOOLS\_DIR and HADOOP\_TOOLS\_LIB\_JARS\_DIR to be consistent with the rest of Hadoop.

• Assignee: Allen Wittenauer
• Reporter: Allen Wittenauer
• Created: 29/Feb/16 22:28
• Updated: 16/Feb/21 08:43
• Resolved: 23/Mar/16 20:48

Description

As hadoop-tools grows bigger and bigger, it's becoming evident that having a single directory that gets sucked in is starting to become a big burden as the number of tools grows. Let's rework this to be smarter.

Issue Links

• blocks: HADOOP-12930 [Umbrella] Dynamic subcommands for hadoop shell scripts
• is related to: HADOOP-17042 Hadoop distcp throws "ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found"
• relates to: HADOOP-13110 add a streaming subcommand to mapred
• relates to: HADOOP-12864 Remove bin/rcc script
• relates to: HADOOP-12865 hadoop-datajoin should be documented or dropped
• relates to: BIGTOP-3495 Optional tools of hadoop-tools should be optional
• relates to: HADOOP-12866 add a subcommand for gridmix
• relates to: HADOOP-12867 clean up how rumen and sls are executed
• relates to: HADOOP-12868 Fix hadoop-openstack undeclared and unused dependencies
• supercedes: HADOOP-12721 Hadoop-tools jars should be included in the classpath of hadoop command