I am not sure I understand the problem that would be solved by making these changes.
The unit tests should always run. The integration tests should fail when things are not set up right.
The testing env should help anyone looking to verify their testing setup. e.g. The test require a temp directory as an Environment Variable. It would be helpful to put things like these into their own tests to verify the setup and help developers figure out what they are missing.
In Lucene.Net 4.8.0, there are no requirements to have any environment variable setup for the tests to run. As long as all of the prerequisites are in place (as specified on the readme https://github.com/apache/lucenenet#building-and-testing), the tests will run.
Am I correct in assuming that this request was a workaround for the fact that Lucene.Net 3.0.3 didn't have the tests setup to "just run" (as they should always be setup)?
We should break down the tests into categories.
- ones that ensure the test env is set up correctly for any integration tests.
- integration tests
- unit tests (ones that use RAMDirectory, in memory objects, etc)
- any others that may be useful.
CI should run these by categories.
As for categorizing tests beyond "long running tests" and "lucene.net specific tests" which we already have, I am not seeing the advantage there either. Nearly all of the tests are integration tests (that is, tests that are setup to test things as a unit instead of one individual component and a bunch of fakes). Putting 80-90% of the 7000+ tests in one big category doesn't seem like it would gain anything. Also, I am not seeing what we gain by running the categories separately in CI. Am I missing something?