[LUCENE-9077] Gradle build

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0
• Lucene Fields: New
• Assignee: Dawid Weiss
• Reporter: Dawid Weiss
• Created: 02/Dec/19 14:13
• Updated: 28/Aug/22 15:53
• Resolved: 29/Sep/20 08:34

Description

This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.

The code lives on gradle-master branch. It is kept with sync with master. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:

gradlew :help

A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.

•  Apply forbiddenAPIs
•  Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
•  Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
• Port other settings and randomizations from common-build.xml
•  Configure security policy/ sandboxing for tests.
•  test's console output on -Ptests.verbose=true
•  add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
• jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
• verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
• Repro-line for failed tests/ runs.
•  add a top-level README note about building with gradle (and the required JVM).
•  add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
• add an equivalent of 'rat-sources' to precommit.
•  add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
• javadoc compilation

Hard-to-implement stuff already investigated:

• (done)  Printing console output of failed tests. There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit tons of output so it's an overkill.
•  (LUCENE-9120) Tests working with security-debug logs or other JVM-early log output. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.

Of lesser importance:

•  Add an equivalent of 'documentation-lint" to precommit.
• Do not require files to be committed before running precommit. (staged files are fine).
• add rendering of javadocs (gradlew javadoc)
• identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
•  Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
• I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution? 
•  There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
• Precommit doesn't catch unused imports
• Attach javadocs to maven publications.
• Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
• if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should  be fixed in the ant build instead.
• Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
• Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).

 

Note: this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:

• Do not try to do too many things at once. If we deviate too far from master, the branch will be hard to merge.
• Do everything in baby-steps and add small, independent build fragments replacing the old ant infrastructure.
• Try to engage people to run, test and contribute early. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.

 

Issue Links

• incorporates: LUCENE-9132 javadoc references source files from classpath entries and fails during compilation
• incorporates: LUCENE-9479 Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build
• incorporates: SOLR-13452 Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
• is a parent of: SOLR-14417 Gradle build sometimes fails RE BlockPoolSlice
• relates to: SOLR-14426 forbidden api error during precommit DateMathFunction
• relates to: SOLR-13914 Umbrella issue for updating the build system to use Gradle
• relates to: SOLR-14033 Fix Hadoop tests with security manager
• relates to: LUCENE-9120 Ability to run gradle tests with security auditing or any other JVM-level messaging
• relates to: LUCENE-9548 Publish master (9.x) snapshots to https://repository.apache.org
• relates to: LUCENE-9505 Gradle tasks with outputs.upToDateWhen {true} are hard to re-run in separation
• relates to: LUCENE-9169 Gradle Wrapper JAR files checksum should be validated
• links to: GitHub Pull Request #1138