[SPARK-27061] Expose Driver UI port on driver service to access UI using service

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Labels: Kubernetes
• Flags: Important
• Assignee: Chandu Kavar
• Reporter: Chandu Kavar
• Created: 05/Mar/19 15:33
• Updated: 17/May/20 18:23
• Resolved: 11/Mar/19 17:41

Description

Currently, we can access the driver logs using 

kubectl port-forward <driver-pod-name> 4040:4040

mentioned in https://spark.apache.org/docs/latest/running-on-kubernetes.html#accessing-driver-ui

We have users who submit spark jobs to Kubernetes, but they don't have access to the cluster. so, they can't user kubectl port-forward command.

If we can expose 4040 port on driver service, we can easily relay these logs to UI using driver service and Nginx reverse proxy.

Issue Links

• links to: GitHub Pull Request #23990