Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9482

DistributedShell job with localization fails in unsecure cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • distributed-shell
    • None

    Description

      DistributedShell job with localization fails in unsecure cluster. The client localizes the input files to home directory (job user) whereas the AM runs as yarn user reads from it's home directory.

      Command:

      yarn jar /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -shell_command ls  -shell_args / -jar  /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -localize_files /tmp/prabhu
      
      Exception in thread "Thread-4" java.io.UncheckedIOException: Error during localization setup
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
      	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
      	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
      	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
      

      Attachments

        1. YARN-9482-001.patch
          7 kB
          Prabhu Joseph
        2. YARN-9482-002.patch
          7 kB
          Prabhu Joseph
        3. YARN-9482-003.patch
          3 kB
          Prabhu Joseph
        4. YARN-9482-004.patch
          3 kB
          Prabhu Joseph

        Issue Links

          Activity

            People

              prabhujoseph Prabhu Joseph
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: