Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.2.0
-
None
-
None
Description
AWS S3 buckets with object lock and a default object retention period (governance/compliance) require the content MD5 header for PUT operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html)
When multipart uploading a blob into such a bucket, a response is returned indicating that the content MD5 header is missing.
The following snippet:
BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3") .endpoint("https://s3.amazonaws.com") .credentials("...", "...") .buildView(BlobStoreContext.class); BlobStore blobStore = blobStoreContext.getBlobStore(); ByteSource payloadByteSource = ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8)); Blob blob = blobStore.blobBuilder("multipart-uploaded-blob") .userMetadata(Collections.emptyMap()) .payload(payloadByteSource) .contentLength(payloadByteSource.size()) .contentMD5(payloadByteSource.hash(Hashing.md5())) .build(); blobStore.putBlob("my-temp-object-locked-bucket", blob, PutOptions.Builder.multipart()); blobStoreContext.close();
Results in the exception:
com.google.common.util.concurrent.UncheckedExecutionException: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=', code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'} at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395) at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349) at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79) at io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53) Caused by: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=', code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'} at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138) at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107) at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91) at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) at com.sun.proxy.$Proxy44.uploadPart(Unknown Source) at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391) at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415) at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at java.lang.Thread.getStackTrace(Thread.java:1559) at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393) ... 25 more
Here are the requests (URL and headers) initiated by JClouds in the above snippet:
GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location {Host=[s3.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]} POST >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads {Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]} PUT >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD {Content-Type=[application/unknown], Content-Length=[11], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=.., SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]} DELETE >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD {Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}
It looks like the blob's MD5 hash is added to the POST to initiate the multipart upload but is missing from the part's PUT itself.