Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build, security
    • Labels:
      None
    • Target Version/s:

      Description

      OWASP's Dependency-Check is a utility that identifies project
      dependencies and checks if there are any known, publicly disclosed,
      vulnerabilities.

      See https://www.owasp.org/index.php/OWASP_Dependency_Check

      This is very useful to stay on top of known vulnerabilities in third party jars. Since it's a maven plugin it's pretty easy to drop in.

        Activity

        Hide
        yoderme Mike Yoder added a comment -
        Show
        yoderme Mike Yoder added a comment - Pinging Andrew Wang , Aaron T. Myers , and Steve Loughran
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM. Do we need precommit or postcommit integration, or is the assumption that someone is running this occasionally and triaging the output?

        Show
        andrew.wang Andrew Wang added a comment - LGTM. Do we need precommit or postcommit integration, or is the assumption that someone is running this occasionally and triaging the output?
        Hide
        yoderme Mike Yoder added a comment -

        (pre|post)commit integration seems rather excessive to me; hopefully third party libraries change slowly. Occasional runs (monthly? per release?) seem fine to me.

        Show
        yoderme Mike Yoder added a comment - (pre|post)commit integration seems rather excessive to me; hopefully third party libraries change slowly. Occasional runs (monthly? per release?) seem fine to me.
        Hide
        lmccay Larry McCay added a comment -

        I think that precommit would be great.
        No one should be able to commit a change that introduces a new vulnerable dependency.

        Would it be possible to make that the criteria? Only block new dependencies that have vulnerabilities?
        Then periodic runs for existing dependencies could have a bounty for showing progress per release or something like that?
        Zero vulnerabilities would be too much especially in the beginning.

        Show
        lmccay Larry McCay added a comment - I think that precommit would be great. No one should be able to commit a change that introduces a new vulnerable dependency. Would it be possible to make that the criteria? Only block new dependencies that have vulnerabilities? Then periodic runs for existing dependencies could have a bounty for showing progress per release or something like that? Zero vulnerabilities would be too much especially in the beginning.
        Hide
        lmccay Larry McCay added a comment -

        I think we might have to be careful about what is published openly as a result of a precommit or even periodic scans, come to think of it. Precommit might be okay if we are blocking it from getting in. We need to discuss with security@a.o to be sure.

        Show
        lmccay Larry McCay added a comment - I think we might have to be careful about what is published openly as a result of a precommit or even periodic scans, come to think of it. Precommit might be okay if we are blocking it from getting in. We need to discuss with security@a.o to be sure.
        Hide
        yoderme Mike Yoder added a comment -

        Another thing to consider with a precommit hook is that the data that dependency-check uses for CVEs is, quite literally, the CVE database. If something pops up there, the results of dependency-check will change shortly thereafter - potentially blocking innocent submittals because suddenly thinks look worse.

        To get serious about things, we'd want to somehow lock down the ability to add new dependencies. IIRC Solr does something with jar signing.

        Show
        yoderme Mike Yoder added a comment - Another thing to consider with a precommit hook is that the data that dependency-check uses for CVEs is, quite literally, the CVE database. If something pops up there, the results of dependency-check will change shortly thereafter - potentially blocking innocent submittals because suddenly thinks look worse. To get serious about things, we'd want to somehow lock down the ability to add new dependencies. IIRC Solr does something with jar signing.
        Hide
        andrew.wang Andrew Wang added a comment -

        The CVE database is public, so publishing the output from this plugin isn't revealing any new info. I think that's fine.

        Let's defer the pre/post commit discussion. I'm happy as long as someone is running this occasionally and looking at the output. Another option I thought of is adding it to the RM checklist. People voting on the release could also check this as while validating the artifacts. See: https://wiki.apache.org/hadoop/HowToRelease

        Overall I'm +1 on this, will commit later unless someone raises some objections. Thanks y'all.

        Show
        andrew.wang Andrew Wang added a comment - The CVE database is public, so publishing the output from this plugin isn't revealing any new info. I think that's fine. Let's defer the pre/post commit discussion. I'm happy as long as someone is running this occasionally and looking at the output. Another option I thought of is adding it to the RM checklist. People voting on the release could also check this as while validating the artifacts. See: https://wiki.apache.org/hadoop/HowToRelease Overall I'm +1 on this, will commit later unless someone raises some objections. Thanks y'all.
        Hide
        andrew.wang Andrew Wang added a comment -

        I'm also setting the target version to 2.8.0 for now. I haven't heard much about further 2.6 or 2.7 releases, but can backport if it becomes relevant.

        Show
        andrew.wang Andrew Wang added a comment - I'm also setting the target version to 2.8.0 for now. I haven't heard much about further 2.6 or 2.7 releases, but can backport if it becomes relevant.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 14s trunk passed
        +1 compile 6m 7s trunk passed
        +1 mvnsite 8m 15s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 javadoc 5m 41s trunk passed
        +1 mvninstall 6m 48s the patch passed
        +1 compile 6m 51s the patch passed
        +1 javac 6m 51s the patch passed
        +1 mvnsite 8m 53s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 5m 35s the patch passed
        -1 unit 132m 30s root in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        189m 19s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.mapreduce.tools.TestCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805988/HADOOP-13198.001.patch
        JIRA Issue HADOOP-13198
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 3f047b8b51c2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 15ed080
        Default Java 1.8.0_91
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/artifact/patchprocess/patch-unit-root.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/testReport/
        modules C: . U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 14s trunk passed +1 compile 6m 7s trunk passed +1 mvnsite 8m 15s trunk passed +1 mvneclipse 0m 36s trunk passed +1 javadoc 5m 41s trunk passed +1 mvninstall 6m 48s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed +1 mvnsite 8m 53s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 5m 35s the patch passed -1 unit 132m 30s root in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 189m 19s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805988/HADOOP-13198.001.patch JIRA Issue HADOOP-13198 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 3f047b8b51c2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 15ed080 Default Java 1.8.0_91 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/testReport/ modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9569/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lmccay Larry McCay added a comment -

        +1 on the patch and for putting in on the RM checklist and it being part of what is tested.
        It should actually be evaluated before publishing an rc.

        Show
        lmccay Larry McCay added a comment - +1 on the patch and for putting in on the RM checklist and it being part of what is tested. It should actually be evaluated before publishing an rc.
        Hide
        andrew.wang Andrew Wang added a comment -

        Great! I've committed this to trunk, branch-2, branch-2.8. Thanks Mike for finding and fixing this, and Larry for discussion and review.

        We need to triage the current plugin output to determine what is safe to ignore. Would one of you be interested in taking this one? Then we can put together a wiki page and add it to the release steps.

        Show
        andrew.wang Andrew Wang added a comment - Great! I've committed this to trunk, branch-2, branch-2.8. Thanks Mike for finding and fixing this, and Larry for discussion and review. We need to triage the current plugin output to determine what is safe to ignore. Would one of you be interested in taking this one? Then we can put together a wiki page and add it to the release steps.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9852 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9852/)
        HADOOP-13198. Add support for OWASP's dependency-check. Contributed by (wang: rev 09b866fd45664ff977702b58b6338ce209729a97)

        • pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9852 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9852/ ) HADOOP-13198 . Add support for OWASP's dependency-check. Contributed by (wang: rev 09b866fd45664ff977702b58b6338ce209729a97) pom.xml
        Hide
        aw Allen Wittenauer added a comment -

        The inability to block out false positives, lack of CVE caching, and HTML output make this functionality less than ideal for any sort of automated job.

        Show
        aw Allen Wittenauer added a comment - The inability to block out false positives, lack of CVE caching, and HTML output make this functionality less than ideal for any sort of automated job.
        Hide
        lmccay Larry McCay added a comment -

        agreed - Allen Wittenauer.

        Show
        lmccay Larry McCay added a comment - agreed - Allen Wittenauer .

          People

          • Assignee:
            yoderme Mike Yoder
            Reporter:
            yoderme Mike Yoder
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development