Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10261

DistributedShell client incorrectly names uploaded script file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0
    • None
    • distributed-shell
    • None
    • Patch

    Description

      When running the DistributedShell using a custom script, it is uploaded to HDFS as ExecScript by the Client. The ApplicationMaster cannot find the script because it tries to download ExecScript.sh.

       

      java.io.FileNotFoundException: File does not exist: hdfs://prod-grunn/user/systems/trainer/application_1584528941904_30608/ExecScript.sh
              at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
              at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)

       

      I have patched the DistributedShell subproject so as the Client uploads it as ExecScript.sh (and ExecScript.bat in case of Windows).

       

      With the change, the ApplicationMaster now correctly runs my custom script.

       

      I'll attach a patch.

      Attachments

        1. YARN-10261.patch
          4 kB
          Markus Jelsma

        Activity

          People

            Unassigned Unassigned
            markus17 Markus Jelsma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: