[TEZ-4084] Tez local mode fails when distributed cache creates link with parent

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: Jacob Tolar
• Reporter: Jacob Tolar
• Created: 11/Sep/19 20:32
• Updated: 15/Nov/19 20:41
• Resolved: 15/Nov/19 20:41

Description

If you configure distributed cache with 'target#parent/link' in Tez Local Mode, the file cannot be created and the job fails:

java.lang.IllegalArgumentException: Invalid prefix or suffix
	at java.nio.file.TempFileHelper.generatePath(TempFileHelper.java:63)
	at java.nio.file.TempFileHelper.create(TempFileHelper.java:127)
	at java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:173)
	at java.nio.file.Files.createTempDirectory(Files.java:950)
	at org.apache.tez.dag.app.launcher.TezLocalCacheManager.localize(TezLocalCacheManager.java:103) 

 

I propose: 

1. Ensure the prefix is always valid (e.g. no path separators in it) when creating the temporary copy of the file in TezLocalCacheManager
2. Update tez local mode to behave the same way as mapreduce local mode in this scenario. Mapreduce local mode also doesn't support these types of links (links with a parent directory specified), but if it encounters them it is a soft failure (WARN log message) not a job failure.

 

It is somewhat trickier to correctly support cache files linked with a nonexistent parent; if that feature is required it can be done as a separate JIRA.