[SPARK-25809] Support additional K8S cluster types for integration tests

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.2, 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Target Version/s: 2.4.1, 3.0.0
• Assignee: Rob Vesse
• Reporter: Rob Vesse
• Created: 23/Oct/18 12:56
• Updated: 17/May/20 18:26
• Resolved: 01/Nov/18 16:34

Description

Currently the Spark on K8S integration tests are hardcoded to use a minikube based backend. It would be nice if developers had more flexibility in the choice of K8S cluster they wish to use for integration testing. More specifically it would be useful to be able to use the built-in Kubernetes support in recent Docker releases and to just use a generic K8S cluster.

Issue Links

• is duplicated by: SPARK-25744 Allow kubernetes integration tests to be run against a real cluster.
• links to: [Github] Pull Request #22805 (rvesse)