Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Problem
End-to-end tests are run in separate cron-<version>-e2e branches. To accomodate the Travis time limits we run a total of 6 jobs each covering a subset of the tests.
These so-called splits are currently managed in the respective branches, and not on master/release branches.
This is a rather hidden detail that not everyone is aware of, nor is it easily discoverable. This has resulted several times in newly added tests not actually being run. Furthermore, if the arguments for tests are modified these changes have to be replicated to each branch.
Proposal
Use jUnit Categories to assign each test explicitly to one of the Travis jobs.
@Category(TravisGroup1.class) public class MyTestRunningInTheFirstJob { ... }
It's a bit on the nose but a rather simple solution.
A given group of tests could be executed by running mvn verify -Dcategories="org.apache.flink.tests.util.TravisGroup1".
All tests can be executed by running mvn verify -Dcategories=""org.apache.flink.tests.util.TravisGroup1""
Future considerations
Tests may furthermore be categorized based on what they are testing (e.g. "Metrics", "Checkpointing", "Kafka") to allow running a certain subset of tests quickly.