[YARN-5306] Yarn should detect and fail fast on duplicate resources in container request

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: yarn
• Assignee: Unassigned
• Reporter: Yesha Vora
• Created: 01/Jul/16 17:52
• Updated: 16/Apr/18 15:55

Description

In some cases, Yarn gets duplicate copies of resources in resource-list.
In this case, you end up with a resource list which contains two copies of resource JAR, with the timestamps of the two separate uploads —only one of which (the later one) is correct. At download time, the NM goes through the list and fails the download when it gets to the one with the older timestamp.

We need some utility class to do a scan & check could be used by the NM at download time (so fail with meaningful errors), and the yarn client could perhaps do the check before launch.

Issue Links

• relates to: SPARK-14423 Handle jar conflict issue when uploading to distributed cache