[SPARK-24964] Please add OWASP Dependency Check to all comonent builds(pom.xml)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Build, MLlib, Spark Core, SparkR
• Labels: build bulk-closed easy-fix security

Environment

All development, build, test, environments.

~/workspace/spark-2.3.1/pom.xml
~/workspace/spark-2.3.1/assembly/pom.xml
~/workspace/spark-2.3.1/common/kvstore/dependency-reduced-pom.xml
~/workspace/spark-2.3.1/common/kvstore/pom.xml
~/workspace/spark-2.3.1/common/network-common/dependency-reduced-pom.xml
~/workspace/spark-2.3.1/common/network-common/pom.xml
~/workspace/spark-2.3.1/common/network-shuffle/dependency-reduced-pom.xml
~/workspace/spark-2.3.1/common/network-shuffle/pom.xml
~/workspace/spark-2.3.1/common/network-yarn/pom.xml
~/workspace/spark-2.3.1/common/sketch/dependency-reduced-pom.xml
~/workspace/spark-2.3.1/common/sketch/pom.xml
~/workspace/spark-2.3.1/common/tags/dependency-reduced-pom.xml
~/workspace/spark-2.3.1/common/tags/pom.xml
~/workspace/spark-2.3.1/common/unsafe/pom.xml
~/workspace/spark-2.3.1/core/pom.xml
~/workspace/spark-2.3.1/examples/pom.xml
~/workspace/spark-2.3.1/external/docker-integration-tests/pom.xml
~/workspace/spark-2.3.1/external/flume/pom.xml
~/workspace/spark-2.3.1/external/flume-assembly/pom.xml
~/workspace/spark-2.3.1/external/flume-sink/pom.xml
~/workspace/spark-2.3.1/external/kafka-0-10/pom.xml
~/workspace/spark-2.3.1/external/kafka-0-10-assembly/pom.xml
~/workspace/spark-2.3.1/external/kafka-0-10-sql/pom.xml
~/workspace/spark-2.3.1/external/kafka-0-8/pom.xml
~/workspace/spark-2.3.1/external/kafka-0-8-assembly/pom.xml
~/workspace/spark-2.3.1/external/kinesis-asl/pom.xml
~/workspace/spark-2.3.1/external/kinesis-asl-assembly/pom.xml
~/workspace/spark-2.3.1/external/spark-ganglia-lgpl/pom.xml
~/workspace/spark-2.3.1/graphx/pom.xml
~/workspace/spark-2.3.1/hadoop-cloud/pom.xml
~/workspace/spark-2.3.1/launcher/pom.xml
~/workspace/spark-2.3.1/mllib/pom.xml
~/workspace/spark-2.3.1/mllib-local/pom.xml
~/workspace/spark-2.3.1/repl/pom.xml
~/workspace/spark-2.3.1/resource-managers/kubernetes/core/pom.xml
~/workspace/spark-2.3.1/resource-managers/mesos/pom.xml
~/workspace/spark-2.3.1/resource-managers/yarn/pom.xml
~/workspace/spark-2.3.1/sql/catalyst/pom.xml
~/workspace/spark-2.3.1/sql/core/pom.xml
~/workspace/spark-2.3.1/sql/hive/pom.xml
~/workspace/spark-2.3.1/sql/hive-thriftserver/pom.xml
~/workspace/spark-2.3.1/streaming/pom.xml
~/workspace/spark-2.3.1/tools/pom.xml

• Flags: Patch, Important
• Assignee: Unassigned
• Reporter: Albert Baker
• Created: 29/Jul/18 20:39
• Updated: 11/Mar/21 07:19
• Resolved: 08/Oct/19 05:42

Description

OWASP DC makes an outbound REST call to MITRE Common Vulnerabilities & Exposures (CVE) to perform a lookup for each dependant .jar to list any/all known vulnerabilities for each jar. This step is needed because a manual MITRE CVE lookup/check on the main component does not include checking for vulnerabilities in dependant libraries.

OWASP Dependency check : https://www.owasp.org/index.php/OWASP_Dependency_Check has plug-ins for most Java build/make types (ant, maven, ivy, gradle). Also, add the appropriate command to the nightly build to generate a report of all known vulnerabilities in any/all third party libraries/dependencies that get pulled in. example : mvn -Powasp -Dtest=false -DfailIfNoTests=false clean aggregate

Generating this report nightly/weekly will help inform the project's development team if any dependant libraries have a reported known vulneraility. Project teams that keep up with removing vulnerabilities on a weekly basis will help protect businesses that rely on these open source componets.