[HBASE-20332] shaded mapreduce module shouldn't include hadoop

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0-alpha-1, 2.1.0
• Component/s: mapreduce, shading

Release Note

<!-- markdown -->

The `hbase-shaded-mapreduce` artifact no longer include its own copy of Hadoop classes. Users who make use of the artifact via YARN should be able to get these classes from YARN's classpath without having to make any changes.

• Assignee: Sean Busbey
• Reporter: Sean Busbey
• Created: 03/Apr/18 15:03
• Updated: 20/Jun/18 01:47
• Resolved: 18/Jun/18 21:07

Description

AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module

1) Folks expect to run yarn / mr apps via hadoop jar / yarn jar
2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers)
3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of some but not all hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes
4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space

Issue Links

• relates to: HBASE-20749 Upgrade our use of checkstyle to 8.6+