Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.
The code lives on gradle-master branch. It is kept with sync with master. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
Apply forbiddenAPIs
Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
Port other settings and randomizations from common-build.xml
Configure security policy/ sandboxing for tests.
test's console output on -Ptests.verbose=true
add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
Repro-line for failed tests/ runs.
add a top-level README note about building with gradle (and the required JVM).
add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
add an equivalent of 'rat-sources' to precommit.
add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
javadoc compilation
Hard-to-implement stuff already investigated:
(done)
Printing console output of failed tests. There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit tons of output so it's an overkill.(LUCENE-9120) Tests working with security-debug logs or other JVM-early log output. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.
Of lesser importance:
Add an equivalent of 'documentation-lint" to precommit.
Do not require files to be committed before running precommit. (staged files are fine).
add rendering of javadocs (gradlew javadoc)
identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
- I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution?
There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
Precommit doesn't catch unused imports
- Attach javadocs to maven publications.
Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
- if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should be fixed in the ant build instead.
Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).
Note: this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:
- Do not try to do too many things at once. If we deviate too far from master, the branch will be hard to merge.
- Do everything in baby-steps and add small, independent build fragments replacing the old ant infrastructure.
- Try to engage people to run, test and contribute early. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.
Attachments
Attachments
Issue Links
- incorporates
-
LUCENE-9132 javadoc references source files from classpath entries and fails during compilation
-
- Resolved
-
-
LUCENE-9479 Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build
-
- Closed
-
-
SOLR-13452 Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
-
- Closed
-
- is a parent of
-
SOLR-14417 Gradle build sometimes fails RE BlockPoolSlice
-
- Closed
-
- relates to
-
SOLR-14426 forbidden api error during precommit DateMathFunction
-
- Closed
-
-
SOLR-13914 Umbrella issue for updating the build system to use Gradle
-
- Resolved
-
-
SOLR-14033 Fix Hadoop tests with security manager
-
- Closed
-
-
LUCENE-9120 Ability to run gradle tests with security auditing or any other JVM-level messaging
-
- Open
-
-
LUCENE-9548 Publish master (9.x) snapshots to https://repository.apache.org
-
- Closed
-
-
LUCENE-9505 Gradle tasks with outputs.upToDateWhen {true} are hard to re-run in separation
-
- Closed
-
-
LUCENE-9169 Gradle Wrapper JAR files checksum should be validated
-
- Closed
-
- links to