[MAPREDUCE-7118] Distributed cache conflicts breaks backwards compatability

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0, 3.2.0
• Fix Version/s: 3.2.0, 3.1.1
• Component/s: mrv2
• Target Version/s: 3.2.0, 3.1.1, 3.0.4
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 03/Jul/18 19:44
• Updated: 24/Jul/18 16:16
• Resolved: 19/Jul/18 21:36

Description

MAPREDUCE-4503 made distributed cache conflicts break job submission, but this was quickly downgraded to a warning in MAPREDUCE-4549. Unfortunately the latter did not go into trunk, so the fix is only in 0.23 and 2.x. When Oozie, Pig, and other downstream projects that can occasionally generate distributed cache conflicts move to Hadoop 3.x the workflows that used to work on 0.23 and 2.x no longer function.

Issue Links

• is caused by: MAPREDUCE-4503 Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
• relates to: MAPREDUCE-4549 Distributed cache conflicts breaks backwards compatability