Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8874

Add fixes and workaround for Java 9 Jigsaw (Module System) to Solr tests

    Details

    • Type: Test
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.0
    • Fix Version/s: 6.0, 6.1, master (7.0)
    • Component/s: Tests
    • Labels:

      Description

      We now have one more week to prepare our build for Java 9 Jigsaw. The next Java 9 EA build will now contain the new Java 9 module system. From that time on, it is no longer possible to test Java 9 unless we fix remaining bugs. Currently Solr does not pass at all, because almost every test fails because the RAMUsageEstimator tries to look into objects in static field where the internals were hidden by Java 9:

         [junit4] ERROR   0.00s | SolrRequestParserTest (suite) <<<
         [junit4]    > Throwable #1: java.lang.IllegalStateException: Unable to access 'private final sun.nio.fs.WindowsFileSystem sun.nio.fs
      .WindowsPath.fs' to estimate memory usage
         [junit4]    >        at __randomizedtesting.SeedInfo.seed([C6C2FAD07A66283B]:0)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:602)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.measureSizeOf(RamUsageEstimator.java:545)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.sizeOfAll(RamUsageEstimator.java:387)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.j
      ava:127)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         [junit4]    >        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
         [junit4]    >        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
         [junit4]    >        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
         [junit4]    >        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
         [junit4]    >        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
         [junit4]    >        at java.lang.Thread.run(java.base@9-ea/Thread.java:804)
         [junit4]    > Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make member of class sun.nio.fs.WindowsPath access
      ible:  module java.base does not export sun.nio.fs to unnamed module @436813f3
         [junit4]    >        at sun.reflect.Reflection.throwInaccessibleObjectException(java.base@9-ea/Reflection.java:420)
         [junit4]    >        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(java.base@9-ea/AccessibleObject.java:174)
         [junit4]    >        at java.lang.reflect.Field.checkCanSetAccessible(java.base@9-ea/Field.java:170)
         [junit4]    >        at java.lang.reflect.Field.setAccessible(java.base@9-ea/Field.java:164)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:597)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:594)
         [junit4]    >        at java.security.AccessController.doPrivileged(java.base@9-ea/Native Method)
         [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:594)
         [junit4]    >        ... 13 more
         [junit4] Completed [1/1 (1!)] in 8.46s, 12 tests, 1 error <<< FAILURES!
      

      I have 2 suggestions:

      • for now disable the static heap usage checks for Java 9 (how? See below!)
      • fix the problems! Mainly we have to ensure that all static, non-final fields in tests pointing to Java obects are nulled on tear down (@AfterClass).

      I have no idea how to do the disabling conditionally. Maybe Dawid Weiss can give a hint.

      In addition, almost every Hadoop test fails with Java 9 module system! Unless Hadoop fixes this and we have updated libs, our only chance is to disable all hadoop related stuff with Java 9 preview builds - sorry. I'd like to add a new Test annotation @DisableWithJava9Jigsaw, which disables the test if running under Java 9 Jigsaw.

      1. SOLR-8874.patch
        11 kB
        Uwe Schindler
      2. SOLR-8874.patch
        10 kB
        Uwe Schindler
      3. SOLR-8874.patch
        6 kB
        Uwe Schindler
      4. SOLR-8874.patch
        6 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          thetaphi Uwe Schindler added a comment -

          Preliminary patch:

          • Disables the static leak detector in Solr tests (by package name) if running under Java 9 Jigsaw.
          • Add new annotation and disable the first test: SOLR-8052

          I did not yet run under linux, so Hadoop tests did not run, but all other tests pass now.

          I will figure out the Hadoop situation tomorrow and add the new annotation accordingly. After that I will setup Jenkins to run with Java 9 Jigsaw build 109 (110 is broken).

          Show
          thetaphi Uwe Schindler added a comment - Preliminary patch: Disables the static leak detector in Solr tests (by package name) if running under Java 9 Jigsaw. Add new annotation and disable the first test: SOLR-8052 I did not yet run under linux, so Hadoop tests did not run, but all other tests pass now. I will figure out the Hadoop situation tomorrow and add the new annotation accordingly. After that I will setup Jenkins to run with Java 9 Jigsaw build 109 (110 is broken).
          Hide
          thetaphi Uwe Schindler added a comment -

          Simplified test rule.

          Show
          thetaphi Uwe Schindler added a comment - Simplified test rule.
          Hide
          thetaphi Uwe Schindler added a comment -

          I was able to get hold of the static leak detector issues:

          • I added java.nio.fs.Path and java.io.File to the list of ignored types (the latter because it has a transient reference to Path)
          • I added java.lang.Class to the list, too (it consumes no memory, because they are singletons)
          • I fixed SolrTestCaseJ4 and some other tests to not sit on the core Path for completeness.

          So I removed the hack in LTC. Running tests again...

          Show
          thetaphi Uwe Schindler added a comment - I was able to get hold of the static leak detector issues: I added java.nio.fs.Path and java.io.File to the list of ignored types (the latter because it has a transient reference to Path) I added java.lang.Class to the list, too (it consumes no memory, because they are singletons) I fixed SolrTestCaseJ4 and some other tests to not sit on the core Path for completeness. So I removed the hack in LTC. Running tests again...
          Hide
          thetaphi Uwe Schindler added a comment -

          New patch:

          • I reverted all in LTC, except the excluded types for static leak checker
          • I disabled all HDFS tests through the ant property (like Windows)
          • I disabled the SOLR-8052 test completely for Java 9, it also failed quite often with non-Jigsaw builds

          I will commit this in a moment.

          Show
          thetaphi Uwe Schindler added a comment - New patch: I reverted all in LTC, except the excluded types for static leak checker I disabled all HDFS tests through the ant property (like Windows) I disabled the SOLR-8052 test completely for Java 9, it also failed quite often with non-Jigsaw builds I will commit this in a moment.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f93f90ca1b22d67d738cbd238ef44c292dd82274 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f93f90c ]

          SOLR-8874: Make Solr tests work with Java 9 Jigsaw

          Show
          jira-bot ASF subversion and git services added a comment - Commit f93f90ca1b22d67d738cbd238ef44c292dd82274 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f93f90c ] SOLR-8874 : Make Solr tests work with Java 9 Jigsaw
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0a89ffaf96f63617968c208afcaf9bfb293490ea in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0a89ffa ]

          SOLR-8874: Make Solr tests work with Java 9 Jigsaw

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0a89ffaf96f63617968c208afcaf9bfb293490ea in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0a89ffa ] SOLR-8874 : Make Solr tests work with Java 9 Jigsaw
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7a2bd4318cc98fe612ab97cb37634e44bff8080c in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7a2bd43 ]

          SOLR-8874: Make Solr tests work with Java 9 Jigsaw

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7a2bd4318cc98fe612ab97cb37634e44bff8080c in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7a2bd43 ] SOLR-8874 : Make Solr tests work with Java 9 Jigsaw
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 91424ae9633b2f382799691693dd4ce8ed216cb8 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=91424ae ]

          SOLR-8874, SOLR-8876: Disable more Hadoop tests with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 91424ae9633b2f382799691693dd4ce8ed216cb8 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=91424ae ] SOLR-8874 , SOLR-8876 : Disable more Hadoop tests with Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 19b4168b3f6ef7c6614ece040948cab9a05be32b in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=19b4168 ]

          SOLR-8874, SOLR-8876: Disable more Hadoop tests with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 19b4168b3f6ef7c6614ece040948cab9a05be32b in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=19b4168 ] SOLR-8874 , SOLR-8876 : Disable more Hadoop tests with Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2e06790bdcf0b26fcc8ecf518284432153dd6a7c in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2e06790 ]

          SOLR-8874, SOLR-8876: Disable more Hadoop tests with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2e06790bdcf0b26fcc8ecf518284432153dd6a7c in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2e06790 ] SOLR-8874 , SOLR-8876 : Disable more Hadoop tests with Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3a4e1d114219e0f9a28cf49c51ed9928913d2cb3 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3a4e1d1 ]

          SOLR-8874: Update Maven config to correctly set tests.disableHdfs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3a4e1d114219e0f9a28cf49c51ed9928913d2cb3 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3a4e1d1 ] SOLR-8874 : Update Maven config to correctly set tests.disableHdfs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0f60ce61eb9d28910c5934929d6d300f047ed1ce in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0f60ce6 ]

          SOLR-8874: Update Maven config to correctly set tests.disableHdfs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0f60ce61eb9d28910c5934929d6d300f047ed1ce in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0f60ce6 ] SOLR-8874 : Update Maven config to correctly set tests.disableHdfs
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2ea8d4cd7b39e675ebc2cad6a9bfa983872599a0 in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2ea8d4c ]

          SOLR-8874: Update Maven config to correctly set tests.disableHdfs

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2ea8d4cd7b39e675ebc2cad6a9bfa983872599a0 in lucene-solr's branch refs/heads/branch_6_0 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2ea8d4c ] SOLR-8874 : Update Maven config to correctly set tests.disableHdfs
          Hide
          mdrob Mike Drob added a comment -

          Thanks for figuring this out, Uwe! Hadoop is tracking their Java 9 efforts in HADOOP-11123, looks like there is no activity there for 16 months.

          Show
          mdrob Mike Drob added a comment - Thanks for figuring this out, Uwe! Hadoop is tracking their Java 9 efforts in HADOOP-11123 , looks like there is no activity there for 16 months.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          there's no activity in Hadoop as we tend to be trailing edge: people get upset when we tell them to upgrade. That said: too trailing edge for some of us developers. The JIRA was created as placeholder rather than active dev point.

          1. Nobody is going to be looking @ java 9 right now; java 8 is still getting cleanup details in.
          2. If someone was to do the work for java 9 and supply patches, it can only be beneficial.

          You are probably the first people to test Hadoop and Java 9. What broke?

          Show
          stevel@apache.org Steve Loughran added a comment - there's no activity in Hadoop as we tend to be trailing edge: people get upset when we tell them to upgrade. That said: too trailing edge for some of us developers. The JIRA was created as placeholder rather than active dev point. Nobody is going to be looking @ java 9 right now; java 8 is still getting cleanup details in. If someone was to do the work for java 9 and supply patches, it can only be beneficial. You are probably the first people to test Hadoop and Java 9. What broke?
          Hide
          hossman Hoss Man added a comment -

          Manually correcting fixVersion per Step #S5 of LUCENE-7271

          Show
          hossman Hoss Man added a comment - Manually correcting fixVersion per Step #S5 of LUCENE-7271

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              thetaphi Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development