[LUCENE-10352] Convert TestAllAnalyzersHaveFactories and TestRandomChains to a global integration test and discover classes to check from module system

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.1, 10.0 (main)
• Component/s: modules/analysis
• Labels: random-chains
• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 04/Jan/22 09:25
• Updated: 28/Aug/22 16:33
• Resolved: 05/Jan/22 14:44

Description

Currently TestAllAnalyzersHaveFactories and TestRandomChains only work on the analysis-commons module, but e.g. we do not do a random chain with kuromoji and ICU. Also both tests rely on some hacky classpath-inspection and the tests fail if ran on a JAR file.

This issue tracks progress I am currently doing to refactor this:

• Move those 2 classes to a new gradle subproject :lucene:analysis:integration.tests and add a module-info referring to all other analysis packages
• Rewrite the class discovery to use ModuleReader
• Run TestAllAnalyzersHaveFactories per module (using one module reader), so it discovers all classes and ensures that factory and stream are in same module (there are some core vs. analysis.common discrepancies)
• RunTestRandomChains on the whole module graph. The classes are discovered from all module readers in the graph (filtering on module name starting with "org.apache.lucene.analysis."
• Also compare that the SPI factories returned by discovery match those we have in the module graphs

While doing this I disovered some bad things:

• TestRandomChains depends on test-only resources. We may need to replicate those (it is about 5 files that are fed into the ctors)
• We have 5 different StringMockResourceLoaders: Originally it was only in analysis common, now its everywhere. I will move this class to test-framework. This is unrelated but can be done here. The background of this was that analysis factories and resource loaders were not part of lucene core, so the resourceloader interface couldn't be in test-framework.

Issue Links

• requires: LUCENE-10348 Make stopwords resources from analyzers modules visible to ClasspathResourceLoader and ModuleResourceLoader
• links to: GitHub Pull Request #582