[IGNITE-13209] JavaIgniteCatalogExample doesn't work with a standalone Spark cluster

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.1
• Component/s: spark
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Valentin Kulichenko
• Created: 02/Jul/20 23:57
• Updated: 09/Nov/20 17:28

Description

To reproduce the issue:

1. Start Spark master and slave as described here: http://spark.apache.org/docs/latest/spark-standalone.html
2. Change the master URL in the JavaIgniteCatalogExample from "local" to the one just started.
3. Run the example.

Updated code that creates the IgniteSparkSession:

String libs = "/Users/vkulichenko/GridGain/releases/apache-ignite-2.8.1-bin/libs";
        
IgniteSparkSession igniteSession = IgniteSparkSession.builder()
    .appName("Spark Ignite catalog example")
    .master("spark://Valentin-Kulichenko-MacBook-Pro-1772.local:7077")
    .config("spark.executor.instances", "2")
    .config("spark.executor.extraClassPath", libs + "/*" + ":" + libs + "/ignite-spark/*:" + libs + "/ignite-spring/*")
    .igniteConfig(CONFIG)
    .getOrCreate();

Execution fails with this exception:

[2020-07-02 15:50:27,627][ERROR][task-result-getter-3][TaskSetManager] Task 0 in stage 0.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.0.0.11, executor 0): class org.apache.ignite.IgniteIllegalStateException: Ignite instance with provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite instance? [name=testing]
	at org.apache.ignite.internal.IgnitionEx.grid(IgnitionEx.java:1351)
	at org.apache.ignite.Ignition.ignite(Ignition.java:528)
	at org.apache.ignite.spark.impl.package$.ignite(package.scala:65)
	at org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$2.apply(IgniteRelationProvider.scala:238)
	at org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$2.apply(IgniteRelationProvider.scala:235)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:222)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:144)
	at org.apache.ignite.spark.impl.IgniteSQLDataFrameRDD.compute(IgniteSQLDataFrameRDD.scala:65)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: IGNITE-12397 Apache spark - org.apache.ignite.IgniteIllegalStateException
• relates to: IGNITE-12637 IgniteSparkSession doesn't start the clients on really distributed cluster