[SPARK-28936] Simplify Spark K8s tests by replacing race condition during command execution

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core, Tests
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 31/Aug/19 01:58
• Updated: 17/May/20 18:25
• Resolved: 21/Sep/19 01:19

Description

Currently our command execution for Spark Kubernetes integration tests depends on a Thread.sleep which sometimes doesn't wait long enough. This normally doesn't show up because we automatically retry the the commands inside of an eventually, but on some machines may result in flaky tests.