• Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: master (9.0)
    • Component/s: None
    • Labels:
    • Lucene Fields:


      This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.

      The code lives on gradle-master branch. It is kept with sync with master. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:

      gradlew :help

      A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.

      •  Apply forbiddenAPIs
      •  Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
      •  Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
      • Port other settings and randomizations from common-build.xml
      •  Configure security policy/ sandboxing for tests.
      •  test's console output on -Ptests.verbose=true
      •  add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
      • jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
      • verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
      • Repro-line for failed tests/ runs.
      •  add a top-level README note about building with gradle (and the required JVM).
      •  add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
      • add an equivalent of 'rat-sources' to precommit.
      •  add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
      • javadoc compilation

      Hard-to-implement stuff already investigated:

      • (done)  Printing console output of failed tests. There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit tons of output so it's an overkill.
      •  (LUCENE-9120Tests working with security-debug logs or other JVM-early log output. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.

      Of lesser importance:

      • Add an equivalent of 'documentation-lint" to precommit.
      • Do not require files to be committed before running precommit. (staged files are fine).
      • add rendering of javadocs (gradlew javadoc)
      • Attach javadocs to maven publications.
      • Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
      • if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should  be fixed in the ant build instead.
      • [EOE] identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
      • Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
      • Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).
      • Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
      • I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution? 


      Note: this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:

      • Do not try to do too many things at once. If we deviate too far from master, the branch will be hard to merge.
      • Do everything in baby-steps and add small, independent build fragments replacing the old ant infrastructure.
      • Try to engage people to run, test and contribute early. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.



          Issue Links



              • Assignee:
                dweiss Dawid Weiss
                dweiss Dawid Weiss
              • Votes:
                0 Vote for this issue
                9 Start watching this issue


                • Created:

                  Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0h
                  Time Spent - 4h 50m
                  4h 50m