[HADOOP-15139] [Umbrella] Improvements and fixes for Hadoop shaded client work

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Target Version/s: 3.2.0, 3.1.1
• Assignee: Bharat Viswanadham
• Reporter: Junping Du
• Created: 21/Dec/17 21:55
• Updated: 11/Sep/18 21:34
• Resolved: 11/Sep/18 21:34

Description

In HADOOP-11656, we have made great progress in splitting out third-party dependencies from shaded hadoop client jar (hadoop-client-api), put runtime dependencies in hadoop-client-runtime, and have shaded version of hadoop-client-minicluster for test. However, there are still some left work for this feature to be fully completed:

• We don't have a comprehensive documentation to guide downstream projects/users to use shaded JARs instead of previous JARs
• We should consider to wrap up hadoop tools (distcp, aws, azure) to have shaded version
• More issues could be identified when shaded jars are adopted in more test and production environment, like HADOOP-15137.

Let's have this umbrella JIRA to track all efforts that left to improve hadoop shaded client effort.

CC busbey, bharatviswa and vinodkv.

Issue Links

• relates to: HADOOP-15514 NoClassDefFoundError for TimelineCollectorManager when starting MiniYARNCluster
• relates to: HADOOP-11656 Classpath isolation for downstream clients

Sub-Tasks

• ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster: Resolved
• NoClassDefFoundError for TimelineCollectorManager when starting MiniYARNCluster: Resolved