[ZEPPELIN-5466] Can not specify the spark image from the interpreter settings

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.1, 0.11.0
• Component/s: Kubernetes, spark
• Assignee: rickcheng
• Reporter: rickcheng
• Created: 23/Jul/21 06:44
• Updated: 02/Sep/21 14:25
• Resolved: 02/Sep/21 14:25

Description

When Zeppelin launches a spark interpreter pod under k8s mode, it will build the spark submit options through buildSparkSubmitOptions()

 

But the property "--conf spark.kubernetes.container.image=" will use the value from the "zConf.getK8sSparkContainerImage()". Note that although the interpreter properties overrides the values through

// interpreter properties overrides the values
k8sProperties.putAll(Maps.fromProperties(properties));

 

But the spark submit options has already been built. So if user set the "spark.kubernetes.container.image" from the interpreter settings, it will not set the spark image in the spark submit options.