Lucene - Core
  1. Lucene - Core
  2. LUCENE-6760

TestUtil#randomWhitespace() broken for Java 9 (Unicode 7)

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3.1, 5.4, 6.0
    • Component/s: None
    • Labels:
    • Environment:

      Java 9 build 76+

    • Lucene Fields:
      New

      Description

      Java 9 changed its character tables to unicode version 7.0. The table updates are listed here: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/86206517258d

      Because of this, character \u180E is no longer treated as whitespace, so TestUtil#randomWhitespace() fails.

      I will remove this character from the list and update the documentation JRE_VERSION_MIGRATION.txt.

      We should maybe make this character list dynamic (e.g. TestUtil initializes it on static class init one time by iterating over all 16 bit characters). Maybe somebody else has an idea (there is already a TODO in the code about that).

      1. LUCENE-6760.patch
        2 kB
        Uwe Schindler
      2. LUCENE-6760.patch
        2 kB
        Uwe Schindler
      3. LUCENE-6760.patch
        2 kB
        Uwe Schindler
      4. LUCENE-6760.patch
        1 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          Patch removing the char.

          What do other think, should we make the static array initialized on static init of TestUtil?

          Show
          Uwe Schindler added a comment - Patch removing the char. What do other think, should we make the static array initialized on static init of TestUtil?
          Hide
          Uwe Schindler added a comment -

          This patch makes the list of whitespace chars dynamic. I would prefer this approach, because it is future proof. Currently all our previous Lucene/Solr releases will not pass the tests in Java 9 because of that!

          Show
          Uwe Schindler added a comment - This patch makes the list of whitespace chars dynamic. I would prefer this approach, because it is future proof. Currently all our previous Lucene/Solr releases will not pass the tests in Java 9 because of that!
          Hide
          Uwe Schindler added a comment -

          Better patch using constants.

          Show
          Uwe Schindler added a comment - Better patch using constants.
          Hide
          Robert Muir added a comment -

          Sorry I am against the change, unless lucene supports one and only one JDK version.

          This patch will stop random seeds from reproducing.

          Show
          Robert Muir added a comment - Sorry I am against the change, unless lucene supports one and only one JDK version. This patch will stop random seeds from reproducing.
          Hide
          Uwe Schindler added a comment -

          OK Robert, thanks for the opinion. I will go for the first patch and simply remove the non-whitespace character.

          FYI, I checked with Eclipse call trace: The method is only used by this single test, so it's not really a useful utility method in TestUtil. We should move it to the test. But I will for now only fix the problem, so the recent Java 9 builds pass!

          Show
          Uwe Schindler added a comment - OK Robert, thanks for the opinion. I will go for the first patch and simply remove the non-whitespace character. FYI, I checked with Eclipse call trace: The method is only used by this single test, so it's not really a useful utility method in TestUtil. We should move it to the test. But I will for now only fix the problem, so the recent Java 9 builds pass!
          Hide
          Uwe Schindler added a comment -

          First patch, with improved error message (was unreadable in the failures recently seen). I will commit this in a moment to prevent more test failures.

          Show
          Uwe Schindler added a comment - First patch, with improved error message (was unreadable in the failures recently seen). I will commit this in a moment to prevent more test failures.
          Hide
          Uwe Schindler added a comment -

          Turn sanity check into assert statement.

          Show
          Uwe Schindler added a comment - Turn sanity check into assert statement.
          Hide
          ASF subversion and git services added a comment -

          Commit 1697131 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1697131 ]

          LUCENE-6760: Prevent test failure in Java 9 b76+

          Show
          ASF subversion and git services added a comment - Commit 1697131 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1697131 ] LUCENE-6760 : Prevent test failure in Java 9 b76+
          Hide
          ASF subversion and git services added a comment -

          Commit 1697132 from Uwe Schindler in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1697132 ]

          Merged revision(s) 1697131 from lucene/dev/trunk:
          LUCENE-6760: Prevent test failure in Java 9 b76+

          Show
          ASF subversion and git services added a comment - Commit 1697132 from Uwe Schindler in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1697132 ] Merged revision(s) 1697131 from lucene/dev/trunk: LUCENE-6760 : Prevent test failure in Java 9 b76+
          Hide
          Uwe Schindler added a comment -

          I will open followup issue to move the method to the test

          Show
          Uwe Schindler added a comment - I will open followup issue to move the method to the test
          Hide
          Uwe Schindler added a comment -

          Followup issue is SOLR-7958

          Show
          Uwe Schindler added a comment - Followup issue is SOLR-7958
          Hide
          ASF subversion and git services added a comment -

          Commit 1698227 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1698227 ]

          LUCENE-6760: if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected

          Show
          ASF subversion and git services added a comment - Commit 1698227 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1698227 ] LUCENE-6760 : if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected
          Hide
          ASF subversion and git services added a comment -

          Commit 1698228 from hossman@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1698228 ]

          LUCENE-6760: if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected (merge r1698227)

          Show
          ASF subversion and git services added a comment - Commit 1698228 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1698228 ] LUCENE-6760 : if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected (merge r1698227)
          Hide
          Uwe Schindler added a comment -

          Open for backporting, otherwise we cannot test release with Java 9

          Show
          Uwe Schindler added a comment - Open for backporting, otherwise we cannot test release with Java 9
          Hide
          ASF subversion and git services added a comment -

          Commit 1701816 from Uwe Schindler in branch 'dev/branches/lucene_solr_5_3'
          [ https://svn.apache.org/r1701816 ]

          Backport:
          LUCENE-6760: Prevent test failure in Java 9 b76+
          SOLR-7958: Move TestUtil#randomWhitespace to the only Solr test that is using it
          LUCENE-6760: if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected (merge r1698227)

          Show
          ASF subversion and git services added a comment - Commit 1701816 from Uwe Schindler in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1701816 ] Backport: LUCENE-6760 : Prevent test failure in Java 9 b76+ SOLR-7958 : Move TestUtil#randomWhitespace to the only Solr test that is using it LUCENE-6760 : if the list of whitspace characters changes again in a future JVM, fail fast with a clear error that the test needs corrected – even if jvm assertions are disabled OR we get really lucky with the chars radnomly selected (merge r1698227)
          Hide
          Uwe Schindler added a comment -

          Done.

          Show
          Uwe Schindler added a comment - Done.

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development