Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: None
    • Component/s: build
    • Labels:

      Description

      Update deprecated jetty 6 dependencies, moving forwards to jetty 8. This enables mvn-rpmbuild on Fedora 18 & > platforms.

      1. HADOOP-9650.patch
        58 kB
        Timothy St. Clair
      2. HADOOP-trunk-9650.patch
        58 kB
        Timothy St. Clair

        Issue Links

          Activity

          Hide
          Timothy St. Clair added a comment -

          Updates jetty dependencies to jetty 8.

          Show
          Timothy St. Clair added a comment - Updates jetty dependencies to jetty 8.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12588161/HADOOP-9650.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2658//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588161/HADOOP-9650.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2658//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Adding related jiras.

          Show
          Suresh Srinivas added a comment - Adding related jiras.
          Hide
          Timothy St. Clair added a comment -

          Updating patch against latest branch-2

          Show
          Timothy St. Clair added a comment - Updating patch against latest branch-2
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12588171/HADOOP-9650.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2659//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588171/HADOOP-9650.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2659//console This message is automatically generated.
          Hide
          Timothy St. Clair added a comment -

          In scanning the URL above, I'm inclined to think the auto-builds are incorrect, as I've built against latest branch-2.?.?

          Show
          Timothy St. Clair added a comment - In scanning the URL above, I'm inclined to think the auto-builds are incorrect, as I've built against latest branch-2.?.?
          Hide
          Timothy St. Clair added a comment -

          patch against trunk

          Show
          Timothy St. Clair added a comment - patch against trunk
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12588183/HADOOP-trunk-9650.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified test files.

          -1 javac. The applied patch generated 1173 javac compiler warnings (more than the trunk's current 1152 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-client hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.jmx.TestJMXJsonServlet
          org.apache.hadoop.http.TestSSLHttpServer
          org.apache.hadoop.http.TestHttpServerLifecycle
          org.apache.hadoop.http.TestHttpServer
          org.apache.hadoop.log.TestLogLevel
          org.apache.hadoop.http.TestPathFilter
          org.apache.hadoop.http.TestGlobalFilter
          org.apache.hadoop.http.TestServletFilter
          org.apache.hadoop.http.TestHttpServerWebapps
          org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
          org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
          org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
          org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
          org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestRMStateStore
          org.apache.hadoop.lib.service.hadoop.TestFileSystemAccessService
          org.apache.hadoop.test.TestHTestCase
          org.apache.hadoop.test.TestHFSTestCase
          org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          org.apache.hadoop.fs.http.server.TestHttpFSServer
          org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
          org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588183/HADOOP-trunk-9650.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 23 new or modified test files. -1 javac . The applied patch generated 1173 javac compiler warnings (more than the trunk's current 1152 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-client hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.jmx.TestJMXJsonServlet org.apache.hadoop.http.TestSSLHttpServer org.apache.hadoop.http.TestHttpServerLifecycle org.apache.hadoop.http.TestHttpServer org.apache.hadoop.log.TestLogLevel org.apache.hadoop.http.TestPathFilter org.apache.hadoop.http.TestGlobalFilter org.apache.hadoop.http.TestServletFilter org.apache.hadoop.http.TestHttpServerWebapps org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServer org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater org.apache.hadoop.yarn.server.resourcemanager.recovery.TestRMStateStore org.apache.hadoop.lib.service.hadoop.TestFileSystemAccessService org.apache.hadoop.test.TestHTestCase org.apache.hadoop.test.TestHFSTestCase org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem org.apache.hadoop.fs.http.server.TestHttpFSServer org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2660//console This message is automatically generated.
          Hide
          Timothy St. Clair added a comment -

          shifting sea... fedora 19 has switched to jetty 9.

          Show
          Timothy St. Clair added a comment - shifting sea... fedora 19 has switched to jetty 9.
          Hide
          Alejandro Abdelnur added a comment -

          Timothy St. Clair, typically we move up on versions of dependencies because of a bugfix that is affecting us or because a new feature we want to use.

          Otherwise we stick to the version we have for the following reasons:

          • known stability
          • potential incompatibilities either API or behavior
          • transitive dependencies issues (we need to re-curate all transitive deps we exclude. historically jetty has used their own groupId/artifactId for servlet/jsp, and they've change their names across versions, that is a mess).

          Furthermore, this particular change, is not just an update of the POM, it modifies the code of hadoop as well.

          Also, downstream projects, ie HBase that use stuff from Hadoop built on top of jetty will most likely break.

          So, unless there is something we really need from jetty9 I would be extremely cautious about changing its version when we are trying to get Hadoop 2 stable to claim GA.

          Show
          Alejandro Abdelnur added a comment - Timothy St. Clair , typically we move up on versions of dependencies because of a bugfix that is affecting us or because a new feature we want to use. Otherwise we stick to the version we have for the following reasons: known stability potential incompatibilities either API or behavior transitive dependencies issues (we need to re-curate all transitive deps we exclude. historically jetty has used their own groupId/artifactId for servlet/jsp, and they've change their names across versions, that is a mess). Furthermore, this particular change, is not just an update of the POM, it modifies the code of hadoop as well. Also, downstream projects, ie HBase that use stuff from Hadoop built on top of jetty will most likely break. So, unless there is something we really need from jetty9 I would be extremely cautious about changing its version when we are trying to get Hadoop 2 stable to claim GA.
          Hide
          Timothy St. Clair added a comment -

          Normally I would agree, but Jetty 6 has numerous CVE's against it, I could track them down.

          Show
          Timothy St. Clair added a comment - Normally I would agree, but Jetty 6 has numerous CVE's against it, I could track them down.
          Hide
          Marcelo Vanzin added a comment -

          Hi there, I saw some comments about this change regarding using Jetty 9. I just wanted to point out that, according to Jetty's page, version 9 is Java 7-only. The patches currently attached to the bug bump the version to 8, though, which is fine for Java 6.

          Show
          Marcelo Vanzin added a comment - Hi there, I saw some comments about this change regarding using Jetty 9. I just wanted to point out that, according to Jetty's page, version 9 is Java 7-only. The patches currently attached to the bug bump the version to 8, though, which is fine for Java 6.
          Hide
          Alejandro Abdelnur added a comment -

          Good poing Marcelo, I guess we can only move up to Jetty 8 for now.

          Show
          Alejandro Abdelnur added a comment - Good poing Marcelo, I guess we can only move up to Jetty 8 for now.
          Hide
          Steve Loughran added a comment -

          catching up with these, I'm going to propose that we move up to jetty-8, but with anyone who can offer some load testing of terasort to give it a go just so we can be happy that it's not going to have issues

          Show
          Steve Loughran added a comment - catching up with these, I'm going to propose that we move up to jetty-8, but with anyone who can offer some load testing of terasort to give it a go just so we can be happy that it's not going to have issues
          Hide
          Alejandro Abdelnur added a comment -

          Steve Loughran, Hadoop 2 MR does not use Jetty, uses Netty, thus MR performance should not be affected by this.

          Show
          Alejandro Abdelnur added a comment - Steve Loughran , Hadoop 2 MR does not use Jetty, uses Netty, thus MR performance should not be affected by this.
          Hide
          Guo Ruijing added a comment -

          according to jetty document(http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html)
          1. jetty 9/9.1 is stable
          2. jetty team generally recommend using Jetty 9
          3. jetty 9/9.1 uses java 1.7

          Hadoop 3.0.0 is a good target version since potential incompatibilities either API or behavior as Alejandro's comments.

          for downstream projects, hadoop should warn downstream project to decouple jetty dependency in case of break.

          Show
          Guo Ruijing added a comment - according to jetty document( http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html ) 1. jetty 9/9.1 is stable 2. jetty team generally recommend using Jetty 9 3. jetty 9/9.1 uses java 1.7 Hadoop 3.0.0 is a good target version since potential incompatibilities either API or behavior as Alejandro's comments. for downstream projects, hadoop should warn downstream project to decouple jetty dependency in case of break.

            People

            • Assignee:
              Unassigned
              Reporter:
              Timothy St. Clair
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:

                Development