[SPARK-35084] [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.2, 3.1.1
• Fix Version/s: 3.4.0
• Component/s: Kubernetes
• Assignee: Keunhyun Oh
• Reporter: Keunhyun Oh
• Created: 15/Apr/21 05:25
• Updated: 15/Jan/23 01:49
• Resolved: 15/Jan/23 01:46

Description

I'm trying to migrate spark 2 to spark 3 in k8s.

 

In my environment, on Spark 3.x, jars listed in spark.jars and spark.jars.packages are not added to sparkContext.

After driver's process is launched, jars are not propagated to Executors. So, NoClassDefException is raised in executors.

 

In spark.properties, the only main application jar is contained in spark.jars. It is different from Spark 2.

 

How to solve this situation? Is it any changed spark options in spark 3 from spark 2?

Issue Links

• is related to: SPARK-32545 Applications with Maven dependencies fail when submitted in cluster mode on K8s
• is related to: SPARK-32775 [k8s] Spark client dependency support ignores non-local paths
• is related to: SPARK-33929 Spark-submit with --package deequ doesn't pull all jars
• is related to: SPARK-34870 Jars downloaded with the --packages argument are not added to the classpath for executors.
• is related to: SPARK-23153 Support application dependencies in submission client's local file system
• links to: [Github] Pull Request #32397 (ocworld)
• links to: [Github] Pull Request #32397 (ocworld)
• links to: [Github] Pull Request #38828 (ocworld)