Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9482

DistributedShell job with localization fails in unsecure cluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: distributed-shell
    • Labels:
      None

      Description

      DistributedShell job with localization fails in unsecure cluster. The client localizes the input files to home directory (job user) whereas the AM runs as yarn user reads from it's home directory.

      Command:

      yarn jar /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -shell_command ls  -shell_args / -jar  /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -localize_files /tmp/prabhu
      
      Exception in thread "Thread-4" java.io.UncheckedIOException: Error during localization setup
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
      	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
      	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
      	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
      

        Attachments

        1. YARN-9482-004.patch
          3 kB
          Prabhu Joseph
        2. YARN-9482-003.patch
          3 kB
          Prabhu Joseph
        3. YARN-9482-002.patch
          7 kB
          Prabhu Joseph
        4. YARN-9482-001.patch
          7 kB
          Prabhu Joseph

          Issue Links

            Activity

              People

              • Assignee:
                prabhujoseph Prabhu Joseph
                Reporter:
                prabhujoseph Prabhu Joseph
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: