Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13382

remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Dependencies on commons-httpclient have been removed. Projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Such project are also potentially exposed to the commons-httpclient CVE, and should be fixed for that reason as well.
      Show
      Dependencies on commons-httpclient have been removed. Projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Such project are also potentially exposed to the commons-httpclient CVE, and should be fixed for that reason as well.

      Description

      In branch-2.8 and later, the patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).

      However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files:

      • hadoop-project/pom.xml
      • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

      We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to add the dependency to

      • hadoop-tools/hadoop-openstack/pom.xml
        (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.)
        In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch.

      Other projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Of course that also means such project is exposed to the commons-httpclient CVE, and needs to be fixed for that reason as well.

      1. HADOOP-13382.000.patch
        2 kB
        Matt Foley
      2. HADOOP-13382-branch-2.000.patch
        2 kB
        Matt Foley
      3. HADOOP-13382-branch-2.8.000.patch
        2 kB
        Matt Foley

        Activity

        Hide
        mattf Matt Foley added a comment - - edited

        Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and HADOOP-12552.

        Branch-2 and trunk need exactly the same patch.

        Show
        mattf Matt Foley added a comment - - edited Proposed for branch-2.8, to go along with HADOOP-11613 , HADOOP-12711 , and HADOOP-12552 . Branch-2 and trunk need exactly the same patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 6m 51s trunk passed
        +1 compile 7m 50s trunk passed
        +1 mvnsite 0m 45s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 javadoc 0m 38s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 0m 39s the patch passed
        +1 compile 7m 46s the patch passed
        +1 javac 7m 46s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 3s The patch has no ill-formed XML file.
        +1 javadoc 0m 36s the patch passed
        +1 unit 0m 9s hadoop-project in the patch passed.
        +1 unit 0m 49s hadoop-yarn-registry in the patch passed.
        +1 unit 0m 14s hadoop-openstack in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        30m 15s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818287/HADOOP-13382.000.patch
        JIRA Issue HADOOP-13382
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 0048263e8d78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ea9f437
        Default Java 1.8.0_91
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10012/testReport/
        modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-tools/hadoop-openstack U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10012/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 7m 50s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 36s trunk passed +1 javadoc 0m 38s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 0m 39s the patch passed +1 compile 7m 46s the patch passed +1 javac 7m 46s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 0m 36s the patch passed +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 0m 49s hadoop-yarn-registry in the patch passed. +1 unit 0m 14s hadoop-openstack in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 30m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818287/HADOOP-13382.000.patch JIRA Issue HADOOP-13382 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 0048263e8d78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea9f437 Default Java 1.8.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10012/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-tools/hadoop-openstack U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10012/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hello Matt Foley. +1 for the patch. However, I'm not currently set up to run the hadoop-openstack integration tests to validate. I really need to do that one of these days. In the meantime, Steve Loughran, are you interested in doing another review?

        Show
        cnauroth Chris Nauroth added a comment - Hello Matt Foley . +1 for the patch. However, I'm not currently set up to run the hadoop-openstack integration tests to validate. I really need to do that one of these days. In the meantime, Steve Loughran , are you interested in doing another review?
        Hide
        mattf Matt Foley added a comment - - edited

        Response to the Hadoop QA robot complaint about no new unit tests: This patch seeks to produce no functional change in the behavior of the code, therefore there are no new unit tests needed. There are no new negative tests needed either, because if the patch breaks anything, it will be a gross breakage of hadoop-openstack. Since all existing unit tests continue to work correctly, that's sufficient.

        Show
        mattf Matt Foley added a comment - - edited Response to the Hadoop QA robot complaint about no new unit tests: This patch seeks to produce no functional change in the behavior of the code, therefore there are no new unit tests needed. There are no new negative tests needed either, because if the patch breaks anything, it will be a gross breakage of hadoop-openstack. Since all existing unit tests continue to work correctly, that's sufficient.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        let me do a run of this

        Show
        stevel@apache.org Steve Loughran added a comment - let me do a run of this
        Hide
        stevel@apache.org Steve Loughran added a comment -

        UK networking is a mess; most of the tests are timing out. I'll try later today.

        Show
        stevel@apache.org Steve Loughran added a comment - UK networking is a mess; most of the tests are timing out. I'll try later today.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1

        Show
        stevel@apache.org Steve Loughran added a comment - +1
        Hide
        mattf Matt Foley added a comment -

        Thanks, Chris Nauroth and Steve Loughran. Committed as:

        • trunk - 12aa184479675d6c9bd
        • branch-2 - ea10e1384ff65e27521
        • branch-2.8 - c96cb3fd48925b3eb2c
        Show
        mattf Matt Foley added a comment - Thanks, Chris Nauroth and Steve Loughran . Committed as: trunk - 12aa184479675d6c9bd branch-2 - ea10e1384ff65e27521 branch-2.8 - c96cb3fd48925b3eb2c
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10133/)
        HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM (mfoley: rev 12aa184479675d6c9bd36fd8451f605ee9505b47)

        • hadoop-tools/hadoop-openstack/pom.xml
        • hadoop-project/pom.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10133/ ) HADOOP-13382 . Remove unneeded commons-httpclient dependencies from POM (mfoley: rev 12aa184479675d6c9bd36fd8451f605ee9505b47) hadoop-tools/hadoop-openstack/pom.xml hadoop-project/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
        Hide
        stevel@apache.org Steve Loughran added a comment -

        matt, can you set the fix version? We need this for the automatic change logs

        Show
        stevel@apache.org Steve Loughran added a comment - matt, can you set the fix version? We need this for the automatic change logs
        Hide
        gsaha Gour Saha added a comment -

        Matt Foley this change in branch-2 breaks Apache Slider project which depends on hadoop-common. It is an incompatible change.

        Show
        gsaha Gour Saha added a comment - Matt Foley this change in branch-2 breaks Apache Slider project which depends on hadoop-common. It is an incompatible change.
        Hide
        mattf Matt Foley added a comment -

        Steve Loughran: sorry. Fixed in 2.8.0.
        Gour Saha: It is true that other projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Of course that also means such project is exposed to the commons-httpclient CVE, and needs to be fixed for that reason as well. Will update the Description to note this. Thanks for setting the appropriate flags in the jira.

        Show
        mattf Matt Foley added a comment - Steve Loughran : sorry. Fixed in 2.8.0. Gour Saha : It is true that other projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Of course that also means such project is exposed to the commons-httpclient CVE, and needs to be fixed for that reason as well. Will update the Description to note this. Thanks for setting the appropriate flags in the jira.

          People

          • Assignee:
            mattf Matt Foley
            Reporter:
            mattf Matt Foley
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development