Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23654

Cut jets3t as a dependency of spark-core

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Spark core declares a dependency on Jets3t, which pulls in other cruft

      1. the hadoop-cloud module pulls in the hadoop-aws module with the jets3t-compatible connectors, and the relevant dependencies: the spark-core dependency is incomplete if that module isn't built, and superflous or inconsistent if it is.
      2. We've cut out s3n/s3 and all dependencies on jets3t entirely from hadoop 3.x in favour we're willing to maintain.

      JetS3t was wonderful when it came out, but now the amazon SDKs massively exceed it in functionality, albeit at the expense of week-to-week stability and JAR binary compatibility

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: