[LUCENENET-614] Make Lucene.Net.TestFramework functionality available to end users (as was done in Java)

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Lucene.Net 4.8.0
• Fix Version/s: Lucene.Net 4.8.0
• Component/s: Lucene.Net.TestFramework
• Assignee: Shad Storhaug
• Reporter: Shad Storhaug
• Created: 12/Aug/19 03:00
• Updated: 29/Dec/19 07:58
• Resolved: 29/Dec/19 07:58

Description

Early efforts on Lucene.Net failed to recognize that the test framework was not meant to be an internal component only used by Lucene's tests, but it was also shipped to end users as a Java package so they could use it to test their own custom extensions to Lucene.

We have had at least 2 people request that we make the TestFramework module public.

Challenges:

1. In Java JUnit is ubiquitous, but in .NET there are several test frameworks widely in use. We would need to support at least NUnit, xUnit, and MSTest in order to have wide coverage.
2. Testing frameworks tend to use .NET attributes, and it is not possible to abstract attributes in such a way we can use a single set of attributes to support every test framework.
3. Testing frameworks tend to use static methods to assert test failures, we need to ensure wrapper abstractions are used consistently in Lucene.Net.TestFramework if we are to build an abstraction over each 3rd party test framework's static methods.
4. The API of the test framework wasn't designed with end users in mind. It needs to be cleaned up to use .NET conventions and best practices.
5. The test framework uses abstract classes with tests that are meant to be inherited by the implementer of the subclass. In the past, this has had some strange behavior where the tests were not showing up as belonging to the class library that owned them, but they were shown as "external tests" in Visual Studio Test Explorer.
6. There are no tests that exist to verify the test framework works like it did in Java.

First of all, we should have 3rd party test framework specific adapter assemblies, i.e. Lucene.Net.TestFramework.NUnit, Lucene.Net.TestFramework.xUnit, Lucene.Net.TestFramework.MSTest.

I have done some analysis and it seems that the issue with calling static methods can easily be overcome by consistently using the assertXXX methods that already exist. A wrapper interface could be built to implement these methods, and then in each adapter assembly, an implementation could be created to call into the 3rd-party test framework's static methods.

The existing Java-like methods could be made into internal methods and setup InternalsVisibleToAttribute for all of our test projects so they can still access these methods. The thought is to keep these Java-like so future tests are easy to port, but not to expose them to end users. End users should be directed to use each 3rd party test framework's static methods directly.

For the attributes, there are at least 2 possible options.

1. Quick and dirty: Add conditional compilation around the attributes (for example FEATURE_NUNIT, and use globbing patterns to decide which .cs files from the Lucene.Net.TestFramework folder to include in each package.
2. Extensible:
   1. Rename each end-user-inheritable class in Lucene.Net.TestFramework with the suffix Base.
   2. Create a subclass of each base class in each test framework adapter assembly with the original class name, and add the test framework specific attributes to this class.

The latter approach has the advantage that any 3rd party can make an adapter for use with another 3rd party test framework. The former approach has the advantage that we don't need to ship a Lucene.Net.TestFramwork package in addition to the 3rd party framework adapter packages, and is quicker to implement because we don't need to rename, move or create very many files.

As far as the weird "external tests" behavior, that did not seem to happen with xUnit, and seems to have inconsistent behavior between different versions of NUnit. For NUnit, we either need to check for and patch compatibility with a wide range of versions of NUnit or document that we only support specific versions.

Issue Links

• Dependent: LUCENENET-627 Port RandomizedTesting Test Runner