[SPARK-3431] Parallelize Scala/Java test execution

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Build
• Assignee: Nicholas Chammas
• Reporter: Nicholas Chammas
• Created: 08/Sep/14 00:20
• Updated: 04/Dec/19 03:01
• Resolved: 20/Jan/16 15:53

Description

Running all the tests in dev/run-tests takes up to 2 hours. A common strategy to cut test time down is to parallelize the execution of the tests. Doing that may in turn require some prerequisite changes to be made to how certain tests run.

Issue Links

• is duplicated by: SPARK-2223 Building and running tests with maven is extremely slow
• is related to: SPARK-3490 Alleviate port collisions during tests
• is related to: SPARK-5161 Parallelize Python test execution
• links to: [Github] Pull Request #3564 (nchammas)
• links to: GitHub Pull Request #3564
• links to: Spark dev list: Unit tests in < 5 minutes

Sub-Tasks

• Fix logging of unit test execution time: Resolved