Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 9.0
    • None
    • None
    • New

    Description

      This task focuses on providing gradle-based build equivalent for Lucene and Solr (on master branch). See notes below on why this respin is needed.

      The code lives on gradle-master branch. It is kept with sync with master. Try running the following to see an overview of helper guides concerning typical workflow, testing and ant-migration helpers:

      gradlew :help

      A list of items that needs to be added or requires work. If you'd like to work on any of these, please add your name to the list. Once you have a patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.

      •  Apply forbiddenAPIs
      •  Generate hardware-aware gradle defaults for parallelism (count of workers and test JVMs).
      •  Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level).
      • Port other settings and randomizations from common-build.xml
      •  Configure security policy/ sandboxing for tests.
      •  test's console output on -Ptests.verbose=true
      •  add a :helpDeps explanation to how the dependency system works (palantir plugin, lockfile) and how to retrieve structured information about current dependencies of a given module (in a tree-like output).
      • jar checksums, jar checksum computation and validation. This should be done without intermediate folders (directly on dependency sets).
      • verify min. JVM version and exact gradle version on build startup to minimize odd build side-effects
      • Repro-line for failed tests/ runs.
      •  add a top-level README note about building with gradle (and the required JVM).
      •  add an equivalent of 'validate-source-patterns' (check-source-patterns.groovy) to precommit.
      • add an equivalent of 'rat-sources' to precommit.
      •  add an equivalent of 'check-example-lucene-match-version' (solr only) to precommit.
      • javadoc compilation

      Hard-to-implement stuff already investigated:

      • (done)  Printing console output of failed tests. There doesn't seem to be any way to do this in a reasonably efficient way. There are onOutput listeners but they're slow to operate and solr tests emit tons of output so it's an overkill.
      •  (LUCENE-9120Tests working with security-debug logs or other JVM-early log output. Gradle's test runner works by redirecting Java's stdout/ syserr so this just won't work. Perhaps we can spin the ant-based test runner for such corner-cases.

      Of lesser importance:

      •  Add an equivalent of 'documentation-lint" to precommit.
      • Do not require files to be committed before running precommit. (staged files are fine).
      • add rendering of javadocs (gradlew javadoc)
      • identify and port various "regenerate" tasks from ant builds (javacc, precompiled automata, etc.)
      •  Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently XSLT...)
      • I didn't bother adding Solr dist/test-framework to packaging (who'd use it from a binary distribution? 
      •  There is some python execution in check-broken-links and check-missing-javadocs, not sure if it's been ported
      • Precommit doesn't catch unused imports
      • Attach javadocs to maven publications.
      • Add test 'beasting' (rerunning the same suite multiple times). I'm afraid it'll be difficult to run it sensibly because gradle doesn't offer cwd separation for the forked test runners.
      • if you diff solr packaged distribution against ant-created distribution there are minor differences in library versions and some JARs are excluded/ moved around. I didn't try to force these as everything seems to work (tests, etc.) – perhaps these differences should  be fixed in the ant build instead.
      • Fill in POM details in gradle/defaults-maven.gradle so that they reflect the previous content better (dependencies aside).
      • Add any IDE integration layers that should be added (I use IntelliJ and it imports the project out of the box, without the need for any special tuning).

       

      Note: this builds on the work done by Mark Miller and Cao Mạnh Đạt but also applies lessons learned from those two efforts:

      • Do not try to do too many things at once. If we deviate too far from master, the branch will be hard to merge.
      • Do everything in baby-steps and add small, independent build fragments replacing the old ant infrastructure.
      • Try to engage people to run, test and contribute early. It can't be a one-man effort. The more people understand and can contribute to the build, the more healthy it will be.

       

      Attachments

        Issue Links

          1.
          Build does not work with Java 13 Sub-task Resolved Dawid Weiss  
          2.
          Allow running tests against gradle-unsupported ea-JVMs Sub-task Resolved Dawid Weiss  
          3.
          solr-ref-guide won't build (maven server for gems is down) Sub-task Resolved Dawid Weiss  
          4.
          Port ant-regenerate tasks to Gradle build Sub-task Closed Erick Erickson

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 14h 10m
          5.
          Switch GitHub PR test from ant precommit to gradle Sub-task Resolved Anshum Gupta

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          6.
          Get lint and deprecation warnings out of lucene/queryparsers Sub-task Resolved Erick Erickson  
          7.
          Port Kuromoji dictionary compilation (regenerate) Sub-task Resolved Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          8.
          Port documentation-lint task to Gradle build Sub-task Closed Tomoko Uchida

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4.5h
          9.
          Port ECJ-based linter to gradle Sub-task Closed Tomoko Uchida  
          10.
          Reorganize ./gradlew precommit tasks Sub-task Resolved Erick Erickson  
          11.
          Gradle Javadoc task should output the same documents as Ant Sub-task Closed Tomoko Uchida

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 40m
          12.
          ant nightly-smoke fails due to presence of build.gradle Sub-task Closed Mike Drob

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6.5h
          13.
          Gradle: Jar MANIFEST incomplete Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          14.
          IntelliJ import attempts to compile solr-ref-guide tools/ and fails Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          15.
          Allow builds against arbitrary JVMs (even those unsupported by gradle) Sub-task Closed Dawid Weiss  
          16.
          Incorporate all :precommit tasks into :check Sub-task Closed Dawid Weiss  
          17.
          Port documentation task to gradle Sub-task Closed Uwe Schindler

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10h
          18.
          Migrate validate source patterns task from ant/groovy to gradle/groovy Sub-task Resolved Mike Drob

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          19.
          Port benchmark module Ant build to Gradle Sub-task Closed David Smiley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          20.
          Lucene's gradle version can't cope with Java 14 Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          21.
          Use "release" flag for javac rather than 'source' and 'target' combination Sub-task Reopened Dawid Weiss  
          22.
          Add gradle workflow support for Eclipse IDE Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          23.
          Make an equivalent to Ant's "run" target for Luke module Sub-task Closed Unassigned  
          24.
          Gradle leaves behind junk temporary files Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m

          Activity

            People

              dweiss Dawid Weiss
              dweiss Dawid Weiss
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 48h
                  48h