Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4757

Yarn-client failed to start due to Wrong FS error in distCacheMgr.addResource

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.2.0, 1.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      I got the following error during Spark startup (Yarn-client mode):

      14/12/04 19:33:58 INFO Client: Uploading resource file:/x/home/jianshuang/spark/spark-latest/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar
      java.lang.IllegalArgumentException: Wrong FS: hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar, expected: file:///
      at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
      at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:79)
      at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:506)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
      at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
      at org.apache.spark.deploy.yarn.ClientDistributedCacheManager.addResource(ClientDistributedCacheManager.scala:67)
      at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:257)
      at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:242)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.deploy.yarn.ClientBase$class.prepareLocalResources(ClientBase.scala:242)
      at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:35)
      at org.apache.spark.deploy.yarn.ClientBase$class.createContainerLaunchContext(ClientBase.scala:350)
      at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:35)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:80)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:335)
      at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:986)
      at $iwC$$iwC.<init>(<console>:9)
      at $iwC.<init>(<console>:18)
      at <init>(<console>:20)
      at .<init>(<console>:24)

      According to Liancheng and Andrew, this hotfix might be the root cause:

      https://github.com/apache/spark/commit/38cb2c3a36a5c9ead4494cbc3dde008c2f0698ce

      Jianshi

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              huangjs Jianshi Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: