[SPARK-25682] Docker images generated from dev build and from dist tarball are different

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 08/Oct/18 21:16
• Updated: 17/May/20 18:25
• Resolved: 18/Oct/18 17:22

Description

There's at least one difference I noticed, because of this line:

COPY examples /opt/spark/examples

In a dev build, "examples" contains your usual source code and maven-style directories, whereas in the dist version, it's this:

cp "$SPARK_HOME"/examples/target/scala*/jars/* "$DISTDIR/examples/jars"

So the path to the actual jar files ends up being different depending on how you built the image.

Issue Links

• links to: [Github] Pull Request #22681 (vanzin)
• links to: [Github] Pull Request #22681 (vanzin)