Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-10255

Fully embrace the java module system

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • New

    Description

      I've experimented a bit trying to move the code to the JMS. It is surprisingly difficult... A PoC that almost passes all checks is here:
      -https://github.com/dweiss/lucene/tree/jms-
      https://github.com/dweiss/lucene/tree/jms2

      Here are my conclusions so far:

      • The JMS and gradle add a lot of complexity (this applies to any higher-level tooling, including IDEs, I think). For starters, modules have to be JARs. The effect of this is that what was previously a set of directories from dependencies now has to be a JAR. What was previously an incremental update of a single .class file now ripples throughout the build recreating module JARs (ZIPs!)... I didn't realize it at first, but it's a costly thing to do. I'm not even sure how IDEs handle this issue. (yes, true because gradle splits module resources and classes into separate folders, rendering them unusable as an expanded module).
      • A Java module contains metadata (such as the module version or main class) that is completely detached from any source file. These things live in a class bytecode of the compiled module-info; interestingly, there is no source-level way to specify it - these class attributes are injected by the 'jar' tool. Gradle has some fancy on-the-fly asm conversion filter that injects it.
      • Dependencies between modules will effectively live in two places: in gradle build files and in module-info files. And they can go out of sync, although it's probably easy to catch (since javac would complain about missing classes during compilation, even if they're in module path). (with separate module and classpath configurations there is a possibility to verify the consistency).
      • Probably the biggest challenge (not covered in the PoC) are with our custom javadoc and ecj linter tasks - they see the module-info.java and can't cope with it. At the same time, there is no easy way to exclude that one particular file: ecj would have to accept a full set of sources (command argument limit will be a problem), javac can accept a full set of java sources (external file) but then it doesn't copy doc-files properly anymore (this is probably easier to fix).
      • There are differences at runtime that are hard to anticipate - for example resource lookups via class loader no longer work (I fixed this in Luke).
      • We will have to rethink the long-term strategy of how white-box tests work. There are some guidelines here but all of them have some cons (IDEs being confused). https://docs.gradle.org/current/userguide/java_testing.html#sec:java_testing_modular
      • it's pretty much impossible to exclude transitive dependencies from modules we depend on - if they're not compile-time only (static) requirements, they will have to be present on module path.
      • supporting modules may or may not work in your IDE.

      Attachments

        1. screenshot-2.png
          104 kB
          Uwe Schindler
        2. screenshot-1.png
          75 kB
          Uwe Schindler

        Issue Links

          1.
          Gradle does not write module version attribute for modules with zero dependencies Sub-task Resolved Unassigned  
          2.
          Add a sanity check of consistency between services provided by the modular and classpath layer Sub-task Resolved Dawid Weiss  
          3.
          morfologik-stemming is not an automatic module Sub-task Resolved Unassigned  
          4.
          Module path to dependencies that are not automatic modules (according to gradle) Sub-task Resolved Dawid Weiss  
          5.
          Ukrainian analyzer won't load if started as a module Sub-task Resolved Unassigned  
          6.
          The test-framework as a module (or a separate test-framework-module) Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 8h 20m
          7.
          Make sure IDEs are usable after modules are introduced Sub-task Resolved Unassigned  
          8.
          Tests will fail on expanded modular dependencies that look up resources Sub-task Resolved Unassigned  
          9.
          Add sanity checks for the distribution Sub-task Open Unassigned  
          10.
          Make ecj and javadoc run with modular paths Sub-task Resolved Unassigned  
          11.
          Enable -Xlint:path and -Xlint:-exports Sub-task Open Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          12.
          Modify module descriptors to require core API dependencies and expose them transitively Sub-task Open Unassigned  
          13.
          Automatic cross-validation between the graph of modules and declared dependencies in gradle Sub-task Open Unassigned  
          14.
          Render javadocs of modules as proper modules Sub-task Open Unassigned  
          15.
          gradle emits an empty sourcepath for forked javac and breaks compilation Sub-task Resolved Unassigned  
          16.
          Module path for compiling and running tests is wrong Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 40m
          17.
          Add require derective for jdk.management to core's module descriptor. Sub-task Closed Unassigned

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          18.
          AttributeFactory.DefaultAttributeFactory does not work on module mode Sub-task Resolved Uwe Schindler  
          19.
          ECJ batch compiler does not support --patch-module at all Sub-task Resolved Unassigned  
          20.
          Fix classpath/module path of tests forking their own Java (TestNRTReplication) Sub-task Closed Dawid Weiss

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m

          Activity

            People

              dweiss Dawid Weiss
              dweiss Dawid Weiss
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 58.5h
                  58.5h