[SPARK-29882] SPARK on Kubernetes is Broken for SPARK with no Hadoop release

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.4
• Component/s: Kubernetes, Spark Core

Environment

How was this patch tested?

Kubernetes 1.14, Spark 2.4.4, Hadoop 3.2.1. Adding $SPARK_DIST_CLASSPATH to {{-cp }} param of entrypoint.sh enables launching the executors correctly.

• Flags: Patch
• External issue ID: https://github.com/apache/spark/pull/26493
• Assignee: Unassigned
• Reporter: Shahin Shakeri
• Created: 13/Nov/19 17:03
• Updated: 17/May/20 18:25
• Resolved: 13/Nov/19 18:02

Description

What changes were proposed in this pull request?

Include $SPARK_DIST_CLASSPATH in class path when launching CoarseGrainedExecutorBackend on Kubernetes executors using the provided entrypoint.sh

Why are the changes needed?

For user provided Hadoop (3.2.1 in this example) $SPARK_DIST_CLASSPATH contains the required jars.

Does this PR introduce any user-facing change?

no

 

Issue Links

• duplicates: SPARK-29574 spark with user provided hadoop doesn't work on kubernetes
• links to: GitHub Pull Request #26493