[SPARK-44529] Add a flag to resolve docker tags to hashes at launch time

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.0, 4.0.0
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Holden Karau
• Created: 24/Jul/23 21:28
• Updated: 24/Jul/23 21:28

Description

If you have an Spark docker tag (like say 3.3) you might want to update the container but only for newly launched jobs not existing jobs. To allow this we can resolve the tag to hash at launch time.

 

In some environments this may also offer a small performance improvement as it saves K8s from having to re-resolve the tag with additional executor launches.