[SPARK-24960] k8s: explicitly expose ports on driver container

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.4.0
• Component/s: Deploy, Kubernetes, Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Adelbert Chang
• Created: 28/Jul/18 19:31
• Updated: 17/May/20 17:48
• Resolved: 01/Aug/18 20:59

Description

For the Kubernetes scheduler, the Driver Pod does not explicitly expose its ports. It is possible for a Kubernetes environment to be setup such that Pod ports are closed by default and must be opened explicitly in the Pod spec. In such an environment without this improvement the Driver Service will be unable to route requests (e.g. from the Executors) to the corresponding Driver Pod, which can be observed on the Executor side with this error message:

Caused by: java.io.IOException: Failed to connect to org-apache-spark-examples-sparkpi-1519271450264-driver-svc.dev.svc.cluster.local:7078

 

For posterity, this is a copy of the original issue filed in the now deprecated apache-spark-on-k8s repository.

Issue Links

• links to: [Github] Pull Request #21884 (adelbertc)