Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13044

saveAsTextFile(s3n://) doesn't support s3 Signature Version 4

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.4.0
    • None
    • Input/Output
    • CentOS

    Description

      I have two clusters deployed: US and EU-Frankfort with the same configs on AWS.

      And the application in EU-Frankfort cannot save data to EU-Frankfort-s3, but US one can save to US-s3.

      And I checked and found that EU-Frankfort supports Signature Version 4 only: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

      Code I'm using:

      val s3WriteEndpoint = "s3n://access_key:secret_key@bucket_name/data/12345"
      rdd.saveAsTextFile(s3WriteEndpoint)
      

      So from my issue I guess saveAsTextFile() is using Signature Version 2? How to support Version 4?

      I tried to dig into code
      https://github.com/apache/spark/blob/f14922cff84b1e0984ba4597d764615184126bdc/core/src/main/scala/org/apache/spark/rdd/RDD.scala

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              iamshrek Xin Ren
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: