[SPARK-11353] Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.1, 1.5.1
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Łukasz Piepiora
• Created: 27/Oct/15 16:39
• Updated: 15/Jan/17 15:26
• Resolved: 15/Jan/17 15:26

Description

For certain regions like for example Frankfurt (eu-central-1) AWS supports only AWS Signature Version 4.

Currently Spark is using jets3t library in version 0.9.3, which throws an exception when code tries to save files in S3 in eu-central-1.

Caused by: java.lang.RuntimeException: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@1e4bc601
	at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:238)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:762)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:324)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1143)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1954)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1875)
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1867)
	at org.jets3t.service.StorageService.putObject(StorageService.java:840)
	at org.jets3t.service.S3Service.putObject(S3Service.java:2212)
	at org.jets3t.service.S3Service.putObject(S3Service.java:2356)
	... 23 more
Caused by: java.io.IOException: Stream closed
	at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
	at java.io.BufferedInputStream.reset(BufferedInputStream.java:446)
	at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:236)
	... 33 more

There is a newer version of jets3t 0.9.4, which seems to fix this issue (http://www.jets3t.org/RELEASE_NOTES.html).

Therefore I suggest to upgrade jets3t dependency from 0.9.3 to 0.9.4 for Hadoop profiles.

Issue Links

• duplicates: SPARK-13044 saveAsTextFile(s3n://) doesn't support s3 Signature Version 4
• links to: [Github] Pull Request #9306 (lpiepiora)