[SPARK-9284] Remove tests' dependency on the assembly

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: Tests
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 23/Jul/15 20:30
• Updated: 28/Aug/15 19:34
• Resolved: 28/Aug/15 19:34

Description

Some tests - in particular tests that have to spawn child processes - currently rely on the generated Spark assembly to run properly.

This is sub-optimal for a few reasons:

• Users have to use an unnatural "package everything first, then run tests" approach
• Sometimes tests are run using old code because the user forgot to rebuild the assembly

The latter is particularly annoying in YarnClusterSuite. If you modify some code outside of the yarn/ module, you have to rebuild the whole assembly before that test picks it up.

We should make all tests run without the need to have an assembly around, making sure that they always pick up the latest code compiled by the user.

Issue Links

• links to: [Github] Pull Request #7629 (vanzin)