When we run some job and add a file with alias started with hyphen then a container fails during creating a symlink for a resource file:
yarn jar hadoop-mapreduce-examples.jar pi -files testfile#-symlink 1 1
or add a file to distributed cache in MR job by "job.addCacheFile"
Containers fail if resource file has a symlink started with hyphen with the following error:
Stack trace: ExitCodeException exitCode=1: /tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1537449069809_0022/container_e01_1537449069809_0022_02_000001/launch_container.sh ln: invalid option -- 'y' Try 'ln --help' for more information. at org.apache.hadoop.util.Shell.runCommand(Shell.java:572) at org.apache.hadoop.util.Shell.run(Shell.java:466) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:768) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:306)
The main cause of the problem is "launch_container.sh" script whitch contain the following command for creating a symlink:
ln -sf "/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/filecache/49/testfile" "-symlink"
As the result "-symlink" parse as "-s" flag but not as a symlink name.
The same job successfully passed when running on MRv1 but not on YARN since symlinks create in different ways. Unix systems support names which start with the hyphen.