[SPARK-3534] Avoid running MLlib and Streaming tests when testing SQL PRs

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Project Infra, SQL
• Target Version/s: 1.2.0
• Assignee: Nicholas Chammas
• Reporter: Michael Armbrust
• Created: 15/Sep/14 18:38
• Updated: 18/Sep/14 03:55
• Resolved: 17/Sep/14 19:45

Description

We are bumping up against the 120 minute time limit for tests pretty regularly now. Since we have decreased the number of shuffle partitions and up-ed the parallelism I don't think there is much low hanging fruit to speed up the SQL tests. (The tests that are listed as taking 2-3 minutes are actually 100s of tests that I think are valuable). Instead I propose we avoid running tests that we don't need to.

This will have the added benefit of eliminating failures in SQL due to flaky streaming tests.

Note that this won't fix the full builds that are run for every commit. There I think we just just up the test timeout.

cc: joshrosen pwendell

Issue Links

• duplicates: SPARK-1455 Determine which test suites to run based on code changes
• links to: [Github] Pull Request #2420 (nchammas)
• links to: [Github] Pull Request #2437 (nchammas)
• links to: [Github] Pull Request #2442 (nchammas)