[HADOOP-15248] 400 Bad Request while trying to access S3 through Spark

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 2.7.3
• Component/s: fs/s3

Environment

macOS 10.13.3 (17D47)

Spark 2.2.1

Hadoop 2.7.3

• Assignee: Unassigned
• Reporter: Farshid
• Created: 21/Feb/18 03:44
• Updated: 21/Feb/18 10:54
• Resolved: 21/Feb/18 10:54

Description

 

I'm trying to read a file thorugh s3a from a bucket in us-east-2 (Ohio) and I'm getting 400 Bad Request response:

com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: [removed], AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: [removed]

Since my code works with another bucket in Sydney, it seems to be a signing API version issue (Ohio supports only 4, Sydney supports 2 and 4). So I tried setting the endpoint by adding this to spark-submit as suggested in other posts:

--conf "spark.hadoop.fs.s3a.endpoint=s3.us-east-2.amazonaws.com" 

But that didn't make any difference. I also tried adding the same to a conf file and passing it using --properties-file [file_path]

spark.hadoop.fs.s3a.endpoint               s3.us-east-2.amazonaws.com

No difference. I still get the same error for Ohio (and it doesn't work with Sydney any more, for obvious reasons).