Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8812

Containers fail during creating a symlink which started with hyphen for a resource file

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.4, 3.1.2, 3.3.0, 3.2.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When we run some job and add a file with alias started with hyphen then a container fails during creating a symlink for a resource file:

      yarn jar hadoop-mapreduce-examples.jar pi -files testfile#-symlink  1 1
      

      or add a file to distributed cache in MR job by "job.addCacheFile"

      Containers fail if resource file has a symlink started with hyphen with the following error:

      Stack trace: ExitCodeException exitCode=1: /tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1537449069809_0022/container_e01_1537449069809_0022_02_000001/launch_container.sh
      ln: invalid option -- 'y'
      Try 'ln --help' for more information.
      
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:572)
      at org.apache.hadoop.util.Shell.run(Shell.java:466)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:768)
      at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:306)
      

      The main cause of the problem is "launch_container.sh" script whitch contain the following command for creating a symlink:

      ln -sf "/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/filecache/49/testfile" "-symlink"
      

      As the result "-symlink" parse as "-s" flag but not as a symlink name.

      The same job successfully passed when running on MRv1 but not on YARN since symlinks create in different ways. Unix systems support names which start with the hyphen.

        Attachments

        1. YARN-8812.001.patch
          2 kB
          Oleksandr Shevchenko

          Activity

            People

            • Assignee:
              oshevchenko Oleksandr Shevchenko
              Reporter:
              oshevchenko Oleksandr Shevchenko
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: