Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-13209

JavaIgniteCatalogExample doesn't work with a standalone Spark cluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.1
    • Fix Version/s: None
    • Component/s: spark
    • Labels:
      None
    • Ignite Flags:
      Docs Required, Release Notes Required

      Description

      To reproduce the issue:

      1. Start Spark master and slave as described here: http://spark.apache.org/docs/latest/spark-standalone.html
      2. Change the master URL in the JavaIgniteCatalogExample from "local" to the one just started.
      3. Run the example.

      Updated code that creates the IgniteSparkSession:

      String libs = "/Users/vkulichenko/GridGain/releases/apache-ignite-2.8.1-bin/libs";
              
      IgniteSparkSession igniteSession = IgniteSparkSession.builder()
          .appName("Spark Ignite catalog example")
          .master("spark://Valentin-Kulichenko-MacBook-Pro-1772.local:7077")
          .config("spark.executor.instances", "2")
          .config("spark.executor.extraClassPath", libs + "/*" + ":" + libs + "/ignite-spark/*:" + libs + "/ignite-spring/*")
          .igniteConfig(CONFIG)
          .getOrCreate();
      

      Execution fails with this exception:

      [2020-07-02 15:50:27,627][ERROR][task-result-getter-3][TaskSetManager] Task 0 in stage 0.0 failed 4 times; aborting job
      Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.0.0.11, executor 0): class org.apache.ignite.IgniteIllegalStateException: Ignite instance with provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite instance? [name=testing]
      	at org.apache.ignite.internal.IgnitionEx.grid(IgnitionEx.java:1351)
      	at org.apache.ignite.Ignition.ignite(Ignition.java:528)
      	at org.apache.ignite.spark.impl.package$.ignite(package.scala:65)
      	at org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$2.apply(IgniteRelationProvider.scala:238)
      	at org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$2.apply(IgniteRelationProvider.scala:235)
      	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:222)
      	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:144)
      	at org.apache.ignite.spark.impl.IgniteSQLDataFrameRDD.compute(IgniteSQLDataFrameRDD.scala:65)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      	at org.apache.spark.scheduler.Task.run(Task.scala:109)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vkulichenko Valentin Kulichenko
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: