[SPARK-22634] Update Bouncy castle dependency

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core, SQL, Structured Streaming
• Assignee: Sean R. Owen
• Reporter: Lior Regev
• Created: 28/Nov/17 15:03
• Updated: 09/Aug/18 16:06
• Resolved: 02/Dec/17 13:37

Description

Spark's usage of jets3t library as well as Spark's own Flume and Kafka streaming uses bouncy castle version 1.51
This is an outdated version as the latest one is 1.58
This, in turn renders packages such as spark-hadoopcryptoledger-ds unusable since these require 1.58 and spark's distributions come along with 1.51
My own attempt was to run on EMR, and since I automatically get all of spark's dependecies (bouncy castle 1.51 being one of them) into the classpath, using the library to parse blockchain data failed due to missing functionality.

I have also opened an issue with jets3t to update their dependecy as well, but along with that Spark would have to update it's own or at least be packaged with a newer version

Issue Links

• is related to: SPARK-23654 Cut jets3t as a dependency of spark-core
• is related to: SPARK-23652 Verify error when using ASF s3:// connector. & Jetty 0.9.4
• links to: [Github] Pull Request #19859 (srowen)