Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39969

Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Build
    • None

    Description

      The AWS SDK and matching kinesis versions are now a few iterations behind what is shipping in hadoop 3.3.x. ( see HADOOP-18068 and HADOOP-18344)

      • this updates dependencies/bundling of jackson and httpclient
      • no problems upgrading other than some test regressions

      catching up would be good, as it means that recent s3a releases are not qualified with the AWS SDK release spark is pulling in -and if there is any problem. it'll be a spark team issue.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-42537 Remove obsolete/superfluous imports in spark-hadoop-cloud module

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39863 Upgrade Hadoop to 3.3.4

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-18344 AWS SDK update to 1.12.262 to address jackson CVE-2018-7489 and AWS CVE-2022-31159

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: