Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10966

Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6
    • Fix Version/s: 6.6.1, 6.7, 7.0
    • Component/s: Hadoop Integration, hdfs
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      I did some testing to work around HADOOP-14586 and found a temporary solution. All tests pass with Java 9 build 175 (HDFS, Hadoop Auth / Kerberos).

      This is a temporary workaround until we can upgrade Hadoop, see SOLR-10951

      The trick here is a hack: The Hadoop Shell class tries to parse java.version system property, which is simply "9" on the Java 9 GA / release candidate. It contains no dots and is shorter than 3 characters. Hadoop tries to get the substring(0,3) and fails with an IndexOutOfBoundsException in clinit. To work around this, we do the following on early Solr startup / test startup (in a static analyzer, like we do for logging initialization):

      • set java.version system property to "1.9"
      • initialize the Shell class in Hadoop
      • restore the old value of java.version

      The whole thing is done in a doPrivileged. I ran some tests on Policeman Jenkins, everything works. The hack is only done, if we detect Java 9.

      1. SOLR-10966.patch
        9 kB
        Uwe Schindler
      2. SOLR-10966.patch
        9 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          thetaphi Uwe Schindler added a comment -

          Hack. I will commit this soon to master and 6.x / 6.6, so I can enable testing again!

          Show
          thetaphi Uwe Schindler added a comment - Hack. I will commit this soon to master and 6.x / 6.6, so I can enable testing again!
          Hide
          thetaphi Uwe Schindler added a comment -

          I tested this already on Jenkins, all passes on Java 9: Here is the running test: https://jenkins.thetaphi.de/job/Lucene-Solr-Hadoop-Update/19

          Show
          thetaphi Uwe Schindler added a comment - I tested this already on Jenkins, all passes on Java 9: Here is the running test: https://jenkins.thetaphi.de/job/Lucene-Solr-Hadoop-Update/19
          Hide
          thetaphi Uwe Schindler added a comment -

          Better patch that does not fail if some JVM disallows to update java.version.

          Show
          thetaphi Uwe Schindler added a comment - Better patch that does not fail if some JVM disallows to update java.version .
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 85a27a231fdddb118ee178baac170da0097a02c0 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=85a27a2 ]

          SOLR-10966, HADOOP-14586: Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 85a27a231fdddb118ee178baac170da0097a02c0 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=85a27a2 ] SOLR-10966 , HADOOP-14586 : Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8c7dd72c9a61cbbb81bd5a8bd61b3cd896e89ca0 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8c7dd72 ]

          SOLR-10966, HADOOP-14586: Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8c7dd72c9a61cbbb81bd5a8bd61b3cd896e89ca0 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8c7dd72 ] SOLR-10966 , HADOOP-14586 : Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9
          Hide
          thetaphi Uwe Schindler added a comment -

          Please reopen if we release 6.6.1

          Show
          thetaphi Uwe Schindler added a comment - Please reopen if we release 6.6.1
          Hide
          thetaphi Uwe Schindler added a comment -

          I will backport this to 6.6.1, otheriwse Jenkins doesnt pass.

          Show
          thetaphi Uwe Schindler added a comment - I will backport this to 6.6.1, otheriwse Jenkins doesnt pass.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bab862e98a07daf7ef26e2fc5f7a19171cb3ed86 in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bab862e ]

          SOLR-10966, HADOOP-14586: Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9

          1. Conflicts:
          2. solr/CHANGES.txt
          Show
          jira-bot ASF subversion and git services added a comment - Commit bab862e98a07daf7ef26e2fc5f7a19171cb3ed86 in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bab862e ] SOLR-10966 , HADOOP-14586 : Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9 Conflicts: solr/CHANGES.txt
          Hide
          thetaphi Uwe Schindler added a comment -

          I just noticed: The Hadoop team has released version 2.7.4 to fix this issue. I'd recommend to update (at least in master and 7.x/7.0). I will open an issue.

          Show
          thetaphi Uwe Schindler added a comment - I just noticed: The Hadoop team has released version 2.7.4 to fix this issue. I'd recommend to update (at least in master and 7.x/7.0). I will open an issue.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c68e668dd7b2522f1f6c7785f256882f0fd8875d in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c68e668 ]

          SOLR-11261, SOLR-10966: Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9.
          This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.

          Show
          jira-bot ASF subversion and git services added a comment - Commit c68e668dd7b2522f1f6c7785f256882f0fd8875d in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c68e668 ] SOLR-11261 , SOLR-10966 : Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9. This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dfd42bf8c9dfbafb5f68e644b217acf9981d6555 in lucene-solr's branch refs/heads/branch_7x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=dfd42bf ]

          SOLR-11261, SOLR-10966: Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9.
          This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.

          Show
          jira-bot ASF subversion and git services added a comment - Commit dfd42bf8c9dfbafb5f68e644b217acf9981d6555 in lucene-solr's branch refs/heads/branch_7x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=dfd42bf ] SOLR-11261 , SOLR-10966 : Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9. This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 04c63953cb35b9e921544be7989d2d67a707c159 in lucene-solr's branch refs/heads/branch_7_0 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=04c6395 ]

          SOLR-11261, SOLR-10966: Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9.
          This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 04c63953cb35b9e921544be7989d2d67a707c159 in lucene-solr's branch refs/heads/branch_7_0 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=04c6395 ] SOLR-11261 , SOLR-10966 : Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9. This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c221a596fe23088ae8cee1ff41e7dcf186e3b402 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c221a59 ]

          SOLR-11261, SOLR-10966: Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9.
          This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.

          Show
          jira-bot ASF subversion and git services added a comment - Commit c221a596fe23088ae8cee1ff41e7dcf186e3b402 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c221a59 ] SOLR-11261 , SOLR-10966 : Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9. This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e0b54e6552775e2f71591e772bceb758c8428783 in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e0b54e6 ]

          SOLR-11261, SOLR-10966: Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9.
          This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0.

          1. Conflicts:
          2. solr/CHANGES.txt
          Show
          jira-bot ASF subversion and git services added a comment - Commit e0b54e6552775e2f71591e772bceb758c8428783 in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e0b54e6 ] SOLR-11261 , SOLR-10966 : Upgrade to Hadoop 2.7.4 to fix incompatibility with Java 9. This also reverts commit 85a27a231fdddb118ee178baac170da0097a02c0. Conflicts: solr/CHANGES.txt

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              thetaphi Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development