Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.4.0
    • Component/s: deployment, spark
    • Labels:
      None

      Description

      Deployment settings:

      cat vagrantconfig.yaml
      ...
      memory_size: 4096
      number_cpus: 1
      box: "puppetlabs/centos-7.2-64-nocm"
      repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64"
      num_instances: 1
      distro: centos
      components: [hdfs, yarn, mapreduce, spark]
      enable_local_repo: true
      run_smoke_tests: false
      smoke_test_components: [hdfs, yarn, mapreduce]
      

      After deployed, execute spark commands:

      [root@bigtop1 bin]# ./spark-shell
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
      	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
      	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
      	at scala.Option.getOrElse(Option.scala:121)
      	at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
      	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	... 7 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                evans_ye Evans Ye
                Reporter:
                evans_ye Evans Ye
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: