Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13044

saveAsTextFile(s3n://) doesn't support s3 Signature Version 4

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Input/Output
    • Labels:
    • Environment:

      CentOS

      Description

      I have two clusters deployed: US and EU-Frankfort with the same configs on AWS.

      And the application in EU-Frankfort cannot save data to EU-Frankfort-s3, but US one can save to US-s3.

      And I checked and found that EU-Frankfort supports Signature Version 4 only: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

      Code I'm using:

      val s3WriteEndpoint = "s3n://access_key:secret_key@bucket_name/data/12345"
      rdd.saveAsTextFile(s3WriteEndpoint)
      

      So from my issue I guess saveAsTextFile() is using Signature Version 2? How to support Version 4?

      I tried to dig into code
      https://github.com/apache/spark/blob/f14922cff84b1e0984ba4597d764615184126bdc/core/src/main/scala/org/apache/spark/rdd/RDD.scala

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                iamshrek Xin Ren
              • Votes:
                2 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: