[MAPREDUCE-4493] Distibuted Cache Compatability Issues

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0-alpha1
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: mrv2
• Target Version/s: 0.23.3
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 27/Jul/12 20:56
• Updated: 23/Apr/18 15:24
• Resolved: 31/Jul/12 19:22

Description

The distributed cache does not work like it does in 1.0.

mapreduce.job.cache.symlink.create is completely ignored and symlinks are always created no matter what. Files and archives without a fragment will also have symlinks created.

If two cache archives or cache files happen to have the same name, or same symlink fragment only the last one in the list is localized.

The localCacheArchives and LocalCacheFiles are not set correctly when these duplicates happen causing off by one or more errors for anyone trying to use them.

The reality is that use of symlinking is so common currently that these incompatibilities are not that likely to show up, but we still need to fix them.

Issue Links

• depends upon: MAPREDUCE-4456 LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
• relates to: OOZIE-3227 Eliminate duplicate dependencies when using Hadoop 3 DistributedCache