[SPARK-23654] Cut jets3t as a dependency of spark-core

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Mar/18 16:03
• Updated: 16/Aug/18 19:34
• Resolved: 16/Aug/18 19:34

Description

Spark core declares a dependency on Jets3t, which pulls in other cruft

1. the hadoop-cloud module pulls in the hadoop-aws module with the jets3t-compatible connectors, and the relevant dependencies: the spark-core dependency is incomplete if that module isn't built, and superflous or inconsistent if it is.
2. We've cut out s3n/s3 and all dependencies on jets3t entirely from hadoop 3.x in favour we're willing to maintain.

JetS3t was wonderful when it came out, but now the amazon SDKs massively exceed it in functionality, albeit at the expense of week-to-week stability and JAR binary compatibility

Issue Links

• is related to: SPARK-25111 increment kinesis client/producer lib versions & aws-sdk to match
• is related to: SPARK-23652 Verify error when using ASF s3:// connector. & Jetty 0.9.4
• relates to: SPARK-22634 Update Bouncy castle dependency
• links to: [Github] Pull Request #21146 (steveloughran)
• links to: [Github] Pull Request #22081 (srowen)
• links to: [Github] Pull Request #22117 (steveloughran)