Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25783

Spark shell fails because of jline incompatibility

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Spark Shell
    • Labels:
      None
    • Environment:

      spark 2.4.0-rc3 on hadoop 2.6.0 (cdh 5.15.1) with -Phadoop-provided

      Description

      error i get when launching spark-shell is:

      Spark context Web UI available at http://client:4040
      Spark context available as 'sc' (master = yarn, app id = application_xxx).
      Spark session available as 'spark'.
      Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
      	at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
      	at scala.tools.nsc.interpreter.jline.InteractiveReader.postInit(JLineReader.scala:54)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$1.apply(SparkILoop.scala:190)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$1.apply(SparkILoop.scala:188)
      	at scala.tools.nsc.interpreter.SplashReader.postInit(InteractiveReader.scala:130)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply$mcV$sp(SparkILoop.scala:214)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
      	at scala.tools.nsc.interpreter.ILoop$$anonfun$mumly$1.apply(ILoop.scala:189)
      	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
      	at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:186)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1.org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1(SparkILoop.scala:199)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:267)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:247)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1.withSuppressedSettings$1(SparkILoop.scala:235)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1.startup$1(SparkILoop.scala:247)
      	at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:282)
      	at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
      	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
      	at org.apache.spark.repl.Main$.doMain(Main.scala:78)
      	at org.apache.spark.repl.Main$.main(Main.scala:58)
      	at org.apache.spark.repl.Main.main(Main.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
      	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
      	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

      spark 2.4.0-rc3 which i build with:

      dev/make-distribution.sh --name provided --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0 -Pyarn -Phadoop-provided
      

      and deployed with in spark-env.sh:

      export SPARK_DIST_CLASSPATH=$(hadoop classpath)
      

      hadoop version is 2.6.0 (CDH 5.15.1)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                koert koert kuipers
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: