[SPARK-26597] Support using images with different entrypoints on Kubernetes

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Component/s: Kubernetes, Spark Core
• External issue ID: https://github.com/jupyter/docker-stacks/issues/797
• Assignee: Unassigned
• Reporter: Patrick Clay
• Created: 11/Jan/19 02:16
• Updated: 17/May/20 18:25
• Resolved: 15/Feb/19 20:49

Description

I wish to use arbitrary pre-existing docker images containing Spark with Kubernetes.

Specifically I wish to use jupyter/all-spark-notebook in in-cluster client mode ideally without image modification (I think using images maintained by others is a key advantage of Docker).

It has the full Spark 2.4 binary tarball with Spark's entrypoint.sh in it, but I need to create a child image setting that as the entrypoint to it, because Spark does not let the user specify a k8s command. I needed separate images for kernel / driver and executor, because I need the kernel to have Jupyter's entrypoint. Building and setting the executor image works, but is obnoxious just to set the entrypoint.

The crux of this FR is to add a property for executor (and driver) command to point to entrypoint.sh

I personally don't see why you even have entrypoint.sh instead of making the command be spark-class org.apache.spark.executor.CoarseGrainedExecutorBackend ..., which seems a lot more portable (albeit reliant on the PATH).

Speaking of reliance on PATH it also broke, because they didn't set JAVA_HOME, and install tini from Conda putting it on a different path. These are smaller issues. I'll file an issue on them and try to work them out between here and there.

In general shouldn't Spark on k8s be less coupled to the layout of the image? 

Issue Links

• duplicates: SPARK-24655 [K8S] Custom Docker Image Expectations and Documentation