[SPARK-27460] Running slowest test suites in their own forked JVMs for higher parallelism

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Tests
• Assignee: Gengliang Wang
• Reporter: Xiao Li
• Created: 15/Apr/19 02:38
• Updated: 20/Sep/19 04:04
• Resolved: 18/Apr/19 12:56

Description

We should modify SparkBuild so that the largest / slowest test suites (or collections of suites) can run in their own forked JVMs, allowing them to be run in parallel with each other

Issue Links

• links to: GitHub Pull Request #24373
• links to: GitHub Pull Request #24404
• links to: GitHub Pull Request #24434
• links to: GitHub Pull Request #25861