Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21384

Spark 2.2 + YARN without spark.yarn.jars / spark.yarn.archive fails

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      In making the updated version of Spark 2.2 + YARN it seems that the auto packaging of JARS based on SPARK_HOME isn't quite working (which results in a warning anyways). You can see the build failure in travis at https://travis-ci.org/holdenk/spark-testing-base/builds/252656109 (I've reproed it locally).

      This results in an exception like:

      17/07/12 03:14:11 WARN ResourceLocalizationService: { file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip, 1499829249000, ARCHIVE, null } failed: File file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip does not exist
      java.io.FileNotFoundException: File file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip does not exist
      	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
      	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
      	at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
      	at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      17/07/12 03:14:11 WARN NMAuditLogger: USER=travis	OPERATION=Container Finished - Failed	TARGET=ContainerImpl	RESULT=FAILURE	DESCRIPTION=Container failed with state: LOCALIZATION_FAILED	APPID=application_1499829231193_0001	CONTAINERID=container_1499829231193_0001_01_000001
      17/07/12 03:14:11 WARN DefaultContainerExecutor: delete returned false for path: [/home/travis/build/holdenk/spark-testing-base/target/com.holdenkarau.spark.testing.YARNCluster/com.holdenkarau.spark.testing.YARNCluster-localDir-nm-0_0/usercache/travis/filecache/11]
      17/07/12 03:14:11 WARN DefaultContainerExecutor: delete returned false for path: [/home/travis/build/holdenk/spark-testing-base/target/com.holdenkarau.spark.testing.YARNCluster/com.holdenkarau.spark.testing.YARNCluster-localDir-nm-0_0/usercache/travis/filecache/11_tmp]
      17/07/12 03:14:13 WARN ResourceLocalizationService: { file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip, 1499829249000, ARCHIVE, null } failed: File file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip does not exist
      java.io.FileNotFoundException: File file:/tmp/spark-0dc9dd59-dd7f-48fc-be2c-11a1bbd57d70/__spark_libs__8035392745283841054.zip does not exist
      	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      	at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
      	at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
      	at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
      	at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      The work around of course is to set one of spark.yarn.jars / spark.yarn.archive.

        Attachments

          Activity

            People

            • Assignee:
              devaraj.k Devaraj K
              Reporter:
              holdenk holdenk
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: