[YARN-8812] Containers fail during creating a symlink which started with hyphen for a resource file

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.4, 3.1.2, 3.3.0, 3.2.1
• Hadoop Flags: Reviewed
• Assignee: Oleksandr Shevchenko
• Reporter: Oleksandr Shevchenko
• Created: 21/Sep/18 09:19
• Updated: 28/Nov/18 15:05
• Resolved: 28/Nov/18 14:57

Description

When we run some job and add a file with alias started with hyphen then a container fails during creating a symlink for a resource file:

yarn jar hadoop-mapreduce-examples.jar pi -files testfile#-symlink  1 1

or add a file to distributed cache in MR job by "job.addCacheFile"

Containers fail if resource file has a symlink started with hyphen with the following error:

Stack trace: ExitCodeException exitCode=1: /tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1537449069809_0022/container_e01_1537449069809_0022_02_000001/launch_container.sh
ln: invalid option -- 'y'
Try 'ln --help' for more information.

at org.apache.hadoop.util.Shell.runCommand(Shell.java:572)
at org.apache.hadoop.util.Shell.run(Shell.java:466)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:768)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:306)

The main cause of the problem is "launch_container.sh" script whitch contain the following command for creating a symlink:

ln -sf "/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/filecache/49/testfile" "-symlink"

As the result "-symlink" parse as "-s" flag but not as a symlink name.

The same job successfully passed when running on MRv1 but not on YARN since symlinks create in different ways. Unix systems support names which start with the hyphen.