Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-10255

Fully embrace the java module system


    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • New


      I've experimented a bit trying to move the code to the JMS. It is surprisingly difficult... A PoC that almost passes all checks is here:

      Here are my conclusions so far:

      • The JMS and gradle add a lot of complexity (this applies to any higher-level tooling, including IDEs, I think). For starters, modules have to be JARs. The effect of this is that what was previously a set of directories from dependencies now has to be a JAR. What was previously an incremental update of a single .class file now ripples throughout the build recreating module JARs (ZIPs!)... I didn't realize it at first, but it's a costly thing to do. I'm not even sure how IDEs handle this issue. (yes, true because gradle splits module resources and classes into separate folders, rendering them unusable as an expanded module).
      • A Java module contains metadata (such as the module version or main class) that is completely detached from any source file. These things live in a class bytecode of the compiled module-info; interestingly, there is no source-level way to specify it - these class attributes are injected by the 'jar' tool. Gradle has some fancy on-the-fly asm conversion filter that injects it.
      • Dependencies between modules will effectively live in two places: in gradle build files and in module-info files. And they can go out of sync, although it's probably easy to catch (since javac would complain about missing classes during compilation, even if they're in module path). (with separate module and classpath configurations there is a possibility to verify the consistency).
      • Probably the biggest challenge (not covered in the PoC) are with our custom javadoc and ecj linter tasks - they see the module-info.java and can't cope with it. At the same time, there is no easy way to exclude that one particular file: ecj would have to accept a full set of sources (command argument limit will be a problem), javac can accept a full set of java sources (external file) but then it doesn't copy doc-files properly anymore (this is probably easier to fix).
      • There are differences at runtime that are hard to anticipate - for example resource lookups via class loader no longer work (I fixed this in Luke).
      • We will have to rethink the long-term strategy of how white-box tests work. There are some guidelines here but all of them have some cons (IDEs being confused). https://docs.gradle.org/current/userguide/java_testing.html#sec:java_testing_modular
      • it's pretty much impossible to exclude transitive dependencies from modules we depend on - if they're not compile-time only (static) requirements, they will have to be present on module path.
      • supporting modules may or may not work in your IDE.


        1. screenshot-2.png
          104 kB
          Uwe Schindler
        2. screenshot-1.png
          75 kB
          Uwe Schindler

        Issue Links

          Gradle does not write module version attribute for modules with zero dependencies Sub-task Resolved Unassigned  
          Add a sanity check of consistency between services provided by the modular and classpath layer Sub-task Resolved Dawid Weiss  
          morfologik-stemming is not an automatic module Sub-task Resolved Unassigned  
          Module path to dependencies that are not automatic modules (according to gradle) Sub-task Resolved Dawid Weiss  
          Ukrainian analyzer won't load if started as a module Sub-task Resolved Unassigned  
          The test-framework as a module (or a separate test-framework-module) Sub-task Closed Dawid Weiss


          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 8h 20m
          Make sure IDEs are usable after modules are introduced Sub-task Resolved Unassigned  
          Tests will fail on expanded modular dependencies that look up resources Sub-task Resolved Unassigned  
          Add sanity checks for the distribution Sub-task Open Unassigned  
          Make ecj and javadoc run with modular paths Sub-task Resolved Unassigned  
          Enable -Xlint:path and -Xlint:-exports Sub-task Open Unassigned


          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          Modify module descriptors to require core API dependencies and expose them transitively Sub-task Open Unassigned  
          Automatic cross-validation between the graph of modules and declared dependencies in gradle Sub-task Open Unassigned  
          Render javadocs of modules as proper modules Sub-task Open Unassigned  
          gradle emits an empty sourcepath for forked javac and breaks compilation Sub-task Resolved Unassigned  
          Module path for compiling and running tests is wrong Sub-task Closed Dawid Weiss


          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 40m
          Add require derective for jdk.management to core's module descriptor. Sub-task Closed Unassigned


          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          AttributeFactory.DefaultAttributeFactory does not work on module mode Sub-task Resolved Uwe Schindler  
          ECJ batch compiler does not support --patch-module at all Sub-task Resolved Unassigned  
          Fix classpath/module path of tests forking their own Java (TestNRTReplication) Sub-task Closed Dawid Weiss


          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m



              dweiss Dawid Weiss
              dweiss Dawid Weiss
              0 Vote for this issue
              9 Start watching this issue



                Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0h
                  Time Spent - 58.5h