Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Apache Httpclient has been removed as a dependency. This library is End of Life: people using it should move to its {{httpcore}} successor. If you cannot do that, you must add an explicit dependency on {{httpclient}} in your classpath.

      Description

      httpclient is now end-of-life and is no longer being developed. Now that we have a dependency on httpcore, we should phase out our use of the old discontinued httpclient library in Hadoop. This will allow us to reduce CLASSPATH bloat and get updated code.

      1. HADOOP-10105.part.patch
        5 kB
        Akira Ajisaka
      2. HADOOP-10105.part2.patch
        17 kB
        Akira Ajisaka
      3. HADOOP-10105.patch
        79 kB
        Akira Ajisaka
      4. HADOOP-10105.2.patch
        79 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          The issue was completed but I noticed there is a task to do. Filed HADOOP-14359 to remove unneessary settings to shade commons-httpclient.

          Show
          ajisakaa Akira Ajisaka added a comment - The issue was completed but I noticed there is a task to do. Filed HADOOP-14359 to remove unneessary settings to shade commons-httpclient.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          All sub-tasks are now resolved. Closing this.
          Thanks all for contributing to this issue!

          Show
          ajisakaa Akira Ajisaka added a comment - All sub-tasks are now resolved. Closing this. Thanks all for contributing to this issue!
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          Set target version as 3.0.0-beta1.
          Somehow this incompatible change escaped me. We really need to get this in Hadoop 3 due to multiple security vulnerabilities discovered with httpclient3.

          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited Set target version as 3.0.0-beta1. Somehow this incompatible change escaped me. We really need to get this in Hadoop 3 due to multiple security vulnerabilities discovered with httpclient3.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Guys, due to the security vulnerability issue CVE-2012-5783 http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783, I would highly suggest that we move away from commons-httpclient. At this point, there are still a few uncommitted piece.
          Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Guys, due to the security vulnerability issue CVE-2012-5783 http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783 , I would highly suggest that we move away from commons-httpclient. At this point, there are still a few uncommitted piece. Thanks!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi guys. I filed a new JIRA HADOOP-12710 (Remove dependency on commons-httpclient for TestHttpServerLogs) and list it as a sub-task of this jira.

          Thanks for the effort.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi guys. I filed a new JIRA HADOOP-12710 (Remove dependency on commons-httpclient for TestHttpServerLogs) and list it as a sub-task of this jira. Thanks for the effort.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Tsuyoshi Ozawa. I wrote that on YARN-3217.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Tsuyoshi Ozawa . I wrote that on YARN-3217 .
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Akira Ajisaka, you're right. But I don't think we should revert the change in this case - Hadoop Compatibility guideline doesn't mention any policies about dependency upgrades. Application developers who use AmIpFilter can avoid the issue just by adding old httpclient with the version a dependency. How about writing it on a release note of YARN-3217?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Akira Ajisaka , you're right. But I don't think we should revert the change in this case - Hadoop Compatibility guideline doesn't mention any policies about dependency upgrades. Application developers who use AmIpFilter can avoid the issue just by adding old httpclient with the version a dependency. How about writing it on a release note of YARN-3217 ?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          hadoop-yarn-server-web-proxy is not user-facing.

          There is a class (AmIpFilter) marked as @InterfaceAudience.Public in the module. If a user uses this class and relies on httpclient, the application can fail.

          Show
          ajisakaa Akira Ajisaka added a comment - hadoop-yarn-server-web-proxy is not user-facing. There is a class ( AmIpFilter ) marked as @InterfaceAudience.Public in the module. If a user uses this class and relies on httpclient, the application can fail.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          hadoop-yarn-server-web-proxy is not user-facing. I think we can drop the dependency safely regardless of its incompatibility. I confirmed that MAPREDUCE-6264 is not incompatible change since it doesn't drop any dependencies.

          Show
          ozawa Tsuyoshi Ozawa added a comment - hadoop-yarn-server-web-proxy is not user-facing. I think we can drop the dependency safely regardless of its incompatibility. I confirmed that MAPREDUCE-6264 is not incompatible change since it doesn't drop any dependencies.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          hadoop-yarn-server-web-proxy is not user-facing. I think we can drop the dependency safely regardless of its incompatibility. I confirmed that MAPREDUCE-6264 is not incompatible change since it doesn't drop any dependencies.

          Show
          ozawa Tsuyoshi Ozawa added a comment - hadoop-yarn-server-web-proxy is not user-facing. I think we can drop the dependency safely regardless of its incompatibility. I confirmed that MAPREDUCE-6264 is not incompatible change since it doesn't drop any dependencies.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Marked YARN-3217 as incompatible change. Should we revert it from branch-2 and branch-2.7?

          Show
          ajisakaa Akira Ajisaka added a comment - Marked YARN-3217 as incompatible change. Should we revert it from branch-2 and branch-2.7?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'm thinking YARN-3217 is incompatible change, MAPREDUCE-6264 is not. YARN-3217 drops httpclient dependency.

          Show
          ajisakaa Akira Ajisaka added a comment - I'm thinking YARN-3217 is incompatible change, MAPREDUCE-6264 is not. YARN-3217 drops httpclient dependency.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          MAPREDUCE-6264 and YARN-3217 has been committed already. Should we mark them as incompatible changes? In paricular, I think MAPREDUCE-6264 is a user-facing change.

          Show
          ozawa Tsuyoshi Ozawa added a comment - MAPREDUCE-6264 and YARN-3217 has been committed already. Should we mark them as incompatible changes? In paricular, I think MAPREDUCE-6264 is a user-facing change.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note that if the dependency is cut from httpclient, this must go down as an incompatible change -anything downstream that expected it there is in trouble.

          Show
          stevel@apache.org Steve Loughran added a comment - Note that if the dependency is cut from httpclient, this must go down as an incompatible change -anything downstream that expected it there is in trouble.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          OK, I'll do my best.

          Show
          ozawa Tsuyoshi Ozawa added a comment - OK, I'll do my best.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          it's not good idea to use mixed libraries of http clients.

          We've already been using mixed libraries, therefore I'm thinking we can remove some httpclient dependencies in 2.7.0 releases, and remove the other dependencies in 2.8.0 release.

          Show
          ajisakaa Akira Ajisaka added a comment - it's not good idea to use mixed libraries of http clients. We've already been using mixed libraries, therefore I'm thinking we can remove some httpclient dependencies in 2.7.0 releases, and remove the other dependencies in 2.8.0 release.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Akira Ajisaka Brahma Reddy Battula Thank you for taking these issues. I have one question : what versions are you targeting? If we do this, the timing of upgrading should be at the same time since it's not good idea to use mixed libraries of http clients. I looked some related tickets, and some resolved issues are targeting 2.7.0. Is it possible to remove all dependency at 2.7.0 release? If not, I think it's better to target 2.8.0 release.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Akira Ajisaka Brahma Reddy Battula Thank you for taking these issues. I have one question : what versions are you targeting? If we do this, the timing of upgrading should be at the same time since it's not good idea to use mixed libraries of http clients. I looked some related tickets, and some resolved issues are targeting 2.7.0. Is it possible to remove all dependency at 2.7.0 release? If not, I think it's better to target 2.8.0 release.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Cancelling patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Cancelling patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623613/HADOOP-10105.2.patch
          against trunk revision 1c03376.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5738//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623613/HADOOP-10105.2.patch against trunk revision 1c03376. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5738//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'd like to split this jira to 5.

          • HDFS
          • MapReduce
          • YARN
          • OpenStack
          • Azure
            I'll file jiras.
          Show
          ajisakaa Akira Ajisaka added a comment - I'd like to split this jira to 5. HDFS MapReduce YARN OpenStack Azure I'll file jiras.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I need to run the unit tests of hadoop-openstack project locally because they are skipped in Jenkins. I'll try it.

          Show
          ajisakaa Akira Ajisaka added a comment - I need to run the unit tests of hadoop-openstack project locally because they are skipped in Jenkins. I'll try it.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623613/HADOOP-10105.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3441//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3441//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623613/HADOOP-10105.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3441//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3441//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to fix FindBug warnings and to pass TestJobEndNotifier.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to fix FindBug warnings and to pass TestJobEndNotifier.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623600/HADOOP-10105.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy:

          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
          org.apache.hadoop.mapred.TestJobEndNotifier

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-openstack.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623600/HADOOP-10105.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.mapred.TestJobEndNotifier +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-openstack.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3440//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to remove httpclient dependency except hadoop-openstack project.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to remove httpclient dependency except hadoop-openstack project.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to remove httpclient dependency in WebAppProxyServlet.java. I'll try to remove the dependency in other (more than 10) classes.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to remove httpclient dependency in WebAppProxyServlet.java. I'll try to remove the dependency in other (more than 10) classes.

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development