[SPARK-41392] Add `bouncy-castle` test dependencies to `sql/core` module for Hadoop 3.4.0

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Build
• Labels: pull-request-available
• Assignee: Yang Jie
• Reporter: Steve Loughran
• Created: 05/Dec/22 13:04
• Updated: 24/Mar/24 02:57
• Resolved: 29/Feb/24 15:56

Description

on hadoop trunk (but not the 3.3.x line), spark builds fail with a CNFE

net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile: org/bouncycastle/jce/provider/BouncyCastleProvider

full stack

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile (scala-test-compile-first) on project spark-sql_2.12: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile failed: A required class was missing while executing net.alchim31.maven:scala-maven-plugin:4.7.2:testCompile: org/bouncycastle/jce/provider/BouncyCastleProvider
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>net.alchim31.maven:scala-maven-plugin:4.7.2
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/Users/stevel/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.7.2/scala-maven-plugin-4.7.2.jar
[ERROR] urls[1] = file:/Users/stevel/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.2.0/maven-dependency-tree-3.2.0.jar
[ERROR] urls[2] = file:/Users/stevel/.m2/repository/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar
[ERROR] urls[3] = file:/Users/stevel/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.1.1/maven-reporting-api-3.1.1.jar
[ERROR] urls[4] = file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.11.1/doxia-sink-api-1.11.1.jar
[ERROR] urls[5] = file:/Users/stevel/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.11.1/doxia-logging-api-1.11.1.jar
[ERROR] urls[6] = file:/Users/stevel/.m2/repository/org/apache/maven/maven-archiver/3.6.0/maven-archiver-3.6.0.jar
[ERROR] urls[7] = file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-io/3.4.0/plexus-io-3.4.0.jar
[ERROR] urls[8] = file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
[ERROR] urls[9] = file:/Users/stevel/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
[ERROR] urls[10] = file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-utils/3.4.2/plexus-utils-3.4.2.jar
[ERROR] urls[11] = file:/Users/stevel/.m2/repository/org/codehaus/plexus/plexus-archiver/4.5.0/plexus-archiver-4.5.0.jar
[ERROR] urls[12] = file:/Users/stevel/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar
[ERROR] urls[13] = file:/Users/stevel/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar
[ERROR] urls[14] = file:/Users/stevel/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
[ERROR] urls[15] = file:/Users/stevel/.m2/repository/org/tukaani/xz/1.9/xz-1.9.jar
[ERROR] urls[16] = file:/Users/stevel/.m2/repository/com/github/luben/zstd-jni/1.5.2-4/zstd-jni-1.5.2-4.jar
[ERROR] urls[17] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc_2.13/1.7.1/zinc_2.13-1.7.1.jar
[ERROR] urls[18] = file:/Users/stevel/.m2/repository/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar
[ERROR] urls[19] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-core_2.13/1.7.1/zinc-core_2.13-1.7.1.jar
[ERROR] urls[20] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-apiinfo_2.13/1.7.1/zinc-apiinfo_2.13-1.7.1.jar
[ERROR] urls[21] = file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-bridge_2.13/1.7.1/compiler-bridge_2.13-1.7.1.jar
[ERROR] urls[22] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-classpath_2.13/1.7.1/zinc-classpath_2.13-1.7.1.jar
[ERROR] urls[23] = file:/Users/stevel/.m2/repository/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar
[ERROR] urls[24] = file:/Users/stevel/.m2/repository/org/scala-sbt/compiler-interface/1.7.1/compiler-interface-1.7.1.jar
[ERROR] urls[25] = file:/Users/stevel/.m2/repository/org/scala-sbt/util-interface/1.7.0/util-interface-1.7.0.jar
[ERROR] urls[26] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-persist-core-assembly/1.7.1/zinc-persist-core-assembly-1.7.1.jar
[ERROR] urls[27] = file:/Users/stevel/.m2/repository/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar
[ERROR] urls[28] = file:/Users/stevel/.m2/repository/org/scala-sbt/io_2.13/1.7.0/io_2.13-1.7.0.jar
[ERROR] urls[29] = file:/Users/stevel/.m2/repository/com/swoval/file-tree-views/2.1.9/file-tree-views-2.1.9.jar
[ERROR] urls[30] = file:/Users/stevel/.m2/repository/net/java/dev/jna/jna/5.12.0/jna-5.12.0.jar
[ERROR] urls[31] = file:/Users/stevel/.m2/repository/net/java/dev/jna/jna-platform/5.12.0/jna-platform-5.12.0.jar
[ERROR] urls[32] = file:/Users/stevel/.m2/repository/org/scala-sbt/util-logging_2.13/1.7.0/util-logging_2.13-1.7.0.jar
[ERROR] urls[33] = file:/Users/stevel/.m2/repository/org/scala-sbt/collections_2.13/1.7.0/collections_2.13-1.7.0.jar
[ERROR] urls[34] = file:/Users/stevel/.m2/repository/org/scala-sbt/util-position_2.13/1.7.0/util-position_2.13-1.7.0.jar
[ERROR] urls[35] = file:/Users/stevel/.m2/repository/org/scala-sbt/core-macros_2.13/1.7.0/core-macros_2.13-1.7.0.jar
[ERROR] urls[36] = file:/Users/stevel/.m2/repository/org/jline/jline-terminal/3.19.0/jline-terminal-3.19.0.jar
[ERROR] urls[37] = file:/Users/stevel/.m2/repository/org/scala-lang/scala-reflect/2.13.8/scala-reflect-2.13.8.jar
[ERROR] urls[38] = file:/Users/stevel/.m2/repository/org/scala-sbt/util-relation_2.13/1.7.0/util-relation_2.13-1.7.0.jar
[ERROR] urls[39] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-persist_2.13/1.7.1/zinc-persist_2.13-1.7.1.jar
[ERROR] urls[40] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-compile-core_2.13/1.7.1/zinc-compile-core_2.13-1.7.1.jar
[ERROR] urls[41] = file:/Users/stevel/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.13/1.1.2/scala-parser-combinators_2.13-1.1.2.jar
[ERROR] urls[42] = file:/Users/stevel/.m2/repository/net/openhft/zero-allocation-hashing/0.10.1/zero-allocation-hashing-0.10.1.jar
[ERROR] urls[43] = file:/Users/stevel/.m2/repository/org/scala-sbt/util-control_2.13/1.7.0/util-control_2.13-1.7.0.jar
[ERROR] urls[44] = file:/Users/stevel/.m2/repository/org/scala-sbt/zinc-classfile_2.13/1.7.1/zinc-classfile_2.13-1.7.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[project>org.apache.spark:spark-parent_2.12:3.4.0-SNAPSHOT, parent: ClassRealm[maven.api, parent: null]]]
[ERROR] 
[ERROR] -----------------------------------------------------
[ERROR] : org.bouncycastle.jce.provider.BouncyCastleProvider
[ERROR] -> [Help 1]
[ERROR] 

HADOOP-17563 did upgrade hadoop's import of bouncy castle to 1.68 which did break spark builds earlier (asm, maven shade), but with SPARK-29729 this has gone away. spark will build with the hadoop 3.3.5 RC0 which has that JIRA. Somehow some other hadoop change is triggering this.

Issue Links

• is depended upon by: HADOOP-18540 Upgrade Bouncy Castle to 1.70
• is related to: SPARK-47530 Add `bcpkix-jdk18on` test dependencies to `hive` module for Hadoop 3.4.0
• is related to: HBASE-28295 Few tests are failing due to NCDFE: org/bouncycastle/operator/OperatorCreationException
• relates to: HADOOP-17563 Update Bouncy Castle to 1.68 or later
• relates to: HADOOP-19018 Release Hadoop 3.4.0
• links to: [Github] Pull Request #38974 (LuciferYang)
• links to: [Github] Pull Request #38974 (LuciferYang)
• links to: GitHub Pull Request #38974
• links to: GitHub Pull Request #45317
• links to: https://github.com/apache/spark/pull/38974