[SOLR-8874] Add fixes and workaround for Java 9 Jigsaw (Module System) to Solr tests

• Type: Test
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 6.0
• Fix Version/s: 6.0, 6.1, 7.0
• Component/s: Tests
• Labels: Java9
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 19/Mar/16 00:21
• Updated: 01/Mar/17 02:20
• Resolved: 19/Mar/16 13:56

Description

We now have one more week to prepare our build for Java 9 Jigsaw. The next Java 9 EA build will now contain the new Java 9 module system. From that time on, it is no longer possible to test Java 9 unless we fix remaining bugs. Currently Solr does not pass at all, because almost every test fails because the RAMUsageEstimator tries to look into objects in static field where the internals were hidden by Java 9:

   [junit4] ERROR   0.00s | SolrRequestParserTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.IllegalStateException: Unable to access 'private final sun.nio.fs.WindowsFileSystem sun.nio.fs
.WindowsPath.fs' to estimate memory usage
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([C6C2FAD07A66283B]:0)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:602)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.measureSizeOf(RamUsageEstimator.java:545)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.sizeOfAll(RamUsageEstimator.java:387)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.j
ava:127)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]    >        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]    >        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]    >        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]    >        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]    >        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]    >        at java.lang.Thread.run(java.base@9-ea/Thread.java:804)
   [junit4]    > Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make member of class sun.nio.fs.WindowsPath access
ible:  module java.base does not export sun.nio.fs to unnamed module @436813f3
   [junit4]    >        at sun.reflect.Reflection.throwInaccessibleObjectException(java.base@9-ea/Reflection.java:420)
   [junit4]    >        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(java.base@9-ea/AccessibleObject.java:174)
   [junit4]    >        at java.lang.reflect.Field.checkCanSetAccessible(java.base@9-ea/Field.java:170)
   [junit4]    >        at java.lang.reflect.Field.setAccessible(java.base@9-ea/Field.java:164)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:597)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:594)
   [junit4]    >        at java.security.AccessController.doPrivileged(java.base@9-ea/Native Method)
   [junit4]    >        at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:594)
   [junit4]    >        ... 13 more
   [junit4] Completed [1/1 (1!)] in 8.46s, 12 tests, 1 error <<< FAILURES!

I have 2 suggestions:

• for now disable the static heap usage checks for Java 9 (how? See below!)
• fix the problems! Mainly we have to ensure that all static, non-final fields in tests pointing to Java obects are nulled on tear down (@AfterClass).

I have no idea how to do the disabling conditionally. Maybe Dawid Weiss can give a hint.

In addition, almost every Hadoop test fails with Java 9 module system! Unless Hadoop fixes this and we have updated libs, our only chance is to disable all hadoop related stuff with Java 9 preview builds - sorry. I'd like to add a new Test annotation @DisableWithJava9Jigsaw, which disables the test if running under Java 9 Jigsaw.

Issue Links

• is related to: SOLR-10219 diagnose HDFS test problems with Java9 and/or re-enable these tests
• relates to: SOLR-8052 Tests using MiniKDC do not work with Java 9 Jigsaw
• relates to: SOLR-8876 Morphlines fails with "No command builder registered for ..." when using Java 9 due to morphline "importCommands" config option attempting to resolve classname globs
• requires: HADOOP-11123 Fix Java 9 incompatibilies in Hadoop