Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      Currently 9.3.19.v20170502 is used.

      In hbase 2.0+, 9.4.6.v20170531 is used.

      When starting mini dfs cluster in hbase unit tests, we get the following:

      java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
        at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
        at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:529)
        at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:119)
        at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:949)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:928)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
        at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
      

      This issue is to upgrade Jetty to 9.4 version

      Attachments

        Issue Links

        Activity

          Hi Ted Yu
          Is it because of jetty?
          As I am seeing jetty code 9.4, still I dont see the method getSessionManager?
          https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-server/src/main/java/org/eclipse/jetty/server/session/SessionHandler.java
          Please let me know if i am missing something here.

          bharat Bharat Viswanadham added a comment - Hi Ted Yu Is it because of jetty? As I am seeing jetty code 9.4, still I dont see the method getSessionManager? https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-server/src/main/java/org/eclipse/jetty/server/session/SessionHandler.java Please let me know if i am missing something here.
          yuzhihong@gmail.com Ted Yu added a comment -

          hadoop uses 9.3.19.v20170502

          That was why NoSuchMethodError was encountered when jetty on the classpath is 9.4 (hbase)

          yuzhihong@gmail.com Ted Yu added a comment - hadoop uses 9.3.19.v20170502 That was why NoSuchMethodError was encountered when jetty on the classpath is 9.4 (hbase)

          Ted Yu
          Yup got it.. Thank You for info.
          In 9.4 jetty there is no sessionManager got it.
          will provide a patch for this.

          bharat Bharat Viswanadham added a comment - Ted Yu Yup got it.. Thank You for info. In 9.4 jetty there is no sessionManager got it. will provide a patch for this.
          bharat Bharat Viswanadham added a comment - - edited

          Ted Yu
          Uploaded the patch.
          Followed the guide to upgrade from 9.3 to 9.4
          http://www.eclipse.org/jetty/documentation/9.4.x/upgrading-jetty.html

          bharat Bharat Viswanadham added a comment - - edited Ted Yu Uploaded the patch. Followed the guide to upgrade from 9.3 to 9.4 http://www.eclipse.org/jetty/documentation/9.4.x/upgrading-jetty.html
          yuzhihong@gmail.com Ted Yu added a comment -

          lgtm

          yuzhihong@gmail.com Ted Yu added a comment - lgtm

          Thank You Ted Yu for review.

          bharat Bharat Viswanadham added a comment - Thank You Ted Yu for review.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 14m 25s trunk passed
          +1 compile 16m 2s trunk passed
          +1 checkstyle 2m 7s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 shadedclient 14m 56s branch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 1m 12s trunk passed
                Patch Compile Tests
          0 mvndep 0m 26s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          +1 compile 11m 57s the patch passed
          +1 javac 11m 57s the patch passed
          +1 checkstyle 2m 9s the patch passed
          +1 mvnsite 1m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 shadedclient 10m 14s patch has no errors when building and testing our client artifacts.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 39s the patch passed
          +1 javadoc 1m 19s the patch passed
                Other Tests
          +1 unit 0m 22s hadoop-project in the patch passed.
          -1 unit 8m 29s hadoop-common in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          109m 51s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12890590/HADOOP-14930.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
          uname Linux 6cce048fd8f6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d0046bb
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 14m 25s trunk passed +1 compile 16m 2s trunk passed +1 checkstyle 2m 7s trunk passed +1 mvnsite 1m 22s trunk passed +1 shadedclient 14m 56s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 32s trunk passed +1 javadoc 1m 12s trunk passed       Patch Compile Tests 0 mvndep 0m 26s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 11m 57s the patch passed +1 javac 11m 57s the patch passed +1 checkstyle 2m 9s the patch passed +1 mvnsite 1m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 10m 14s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 39s the patch passed +1 javadoc 1m 19s the patch passed       Other Tests +1 unit 0m 22s hadoop-project in the patch passed. -1 unit 8m 29s hadoop-common in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 109m 51s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12890590/HADOOP-14930.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml uname Linux 6cce048fd8f6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d0046bb Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13459/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          andrew.wang Andrew Wang added a comment -

          How safe is this? John Zhuge / Robert Kanter any thoughts as recent Jetty experts?

          andrew.wang Andrew Wang added a comment - How safe is this? John Zhuge / Robert Kanter any thoughts as recent Jetty experts?
          rkanter Robert Kanter added a comment -

          Jetty moved a number of things, some that we're using and are updated in the patch, from SessionManager to SessionHandler. This means that upgrading Hadoop from Jetty 9.3.x to 9.4.x is going to also force all downstream projects to also upgrade to Jetty 9.4.x. Are we okay with that?

          rkanter Robert Kanter added a comment - Jetty moved a number of things, some that we're using and are updated in the patch, from SessionManager to SessionHandler . This means that upgrading Hadoop from Jetty 9.3.x to 9.4.x is going to also force all downstream projects to also upgrade to Jetty 9.4.x. Are we okay with that?
          andrew.wang Andrew Wang added a comment -

          I'm not okay with that, considering how widely used Jetty is. -1 on at least branch-3.0.

          HBase has a shaded thirdparty module, so this can also be addressed on the HBase side. This approach is discussed on the linked HBASE JIRA. I propose we close this as WONTFIX.

          andrew.wang Andrew Wang added a comment - I'm not okay with that, considering how widely used Jetty is. -1 on at least branch-3.0. HBase has a shaded thirdparty module, so this can also be addressed on the HBase side. This approach is discussed on the linked HBASE JIRA. I propose we close this as WONTFIX.
          aajisaka Akira Ajisaka added a comment -

          Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735?

          aajisaka Akira Ajisaka added a comment - Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735?
          kihwal Kihwal Lee added a comment - - edited

          To clarify, jetty-9.3.20.v20170531 addressed CVE-2017-9735, so the current jetty version (9.3.24.v20180605) in Hadoop 3.x is okay.

          kihwal Kihwal Lee added a comment - - edited To clarify, jetty-9.3.20.v20170531 addressed CVE-2017-9735, so the current jetty version (9.3.24.v20180605) in Hadoop 3.x is okay.
          aajisaka Akira Ajisaka added a comment -

          Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735?

          This is my mistake. Thanks Kihwal Lee for the information.

          aajisaka Akira Ajisaka added a comment - Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735? This is my mistake. Thanks Kihwal Lee for the information.
          aajisaka Akira Ajisaka added a comment -

          Now Jetty 9.3 is EoL, so let's upgrade to 9.4.

          aajisaka Akira Ajisaka added a comment - Now Jetty 9.3 is EoL, so let's upgrade to 9.4.
          kihwal Kihwal Lee added a comment -

          Setting 3.3.0 as target.

          kihwal Kihwal Lee added a comment - Setting 3.3.0 as target.
          smeng Siyao Meng added a comment -

          Closing this one since latest work is being done in HADOOP-16152.

          smeng Siyao Meng added a comment - Closing this one since latest work is being done in HADOOP-16152 .

          People

            bharat Bharat Viswanadham Assign to me
            yuzhihong@gmail.com Ted Yu
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack