[MAPREDUCE-5951] Add support for the YARN Shared Cache

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0
• Labels: BB2015-05-TBR

Release Note

MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.

• Assignee: Chris Trezzo
• Reporter: Chris Trezzo
• Created: 30/Jun/14 21:12
• Updated: 12/Oct/17 18:42
• Resolved: 12/Oct/17 18:35

Description

Implement the necessary changes so that the MapReduce application can leverage the new YARN shared cache (i.e. YARN-1492).

Specifically, allow per-job configuration so that MapReduce jobs can specify which set of resources they would like to cache (i.e. jobjar, libjars, archives, files).

Issue Links

• depends upon: YARN-3637 Handle localization sym-linking correctly at the YARN level
• depends upon: YARN-2217 Shared cache client side changes
• depends upon: YARN-1492 truly shared cache for jars (jobjar/libjar)
• depends upon: MAPREDUCE-6267 Refactor JobSubmitter#copyAndConfigureFiles into it's own class
• is duplicated by: MAPREDUCE-6113 An ability to override LocalResourceVisibility for public resources
• is related to: MAPREDUCE-6824 TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
• is related to: MAPREDUCE-6365 Refactor JobResourceUploader#uploadFilesInternal
• relates to: YARN-5727 Improve YARN shared cache support for LinuxContainerExecutor