[JCLOUDS-1368] GCS cannot upload multipart file larger than 1 GB

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.3
• Fix Version/s: 2.1.0, 2.1.1
• Component/s: jclouds-blobstore
• Labels: google-cloud-storage
• Assignee: Andrew Gaul
• Reporter: Andrew Gaul
• Created: 04/Jan/18 08:54
• Updated: 13/Apr/22 07:04
• Resolved: 09/Jan/18 20:59

Description

When uploading a file:

long size = 33L * (32L * 1024L * 1024L);
ByteSource byteSource = TestUtils.randomByteSource().slice(0, size);
blobStore.createContainerInLocation(null, containerName);
Blob blob = blobStore.blobBuilder(blobName)
      .payload(byteSource)
      .contentLength(size)
      .build();
blobStore.putBlob(containerName, blob, PutOptions.Builder.multipart());

I see an error:

org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/gaul-blobstore0/o/large-blob/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "invalid",
    "message": "The number of source components provided (33) exceeds the maximum (32)"
   }
  ],
  "code": 400,
  "message": "The number of source components provided (33) exceeds the maximum (32)"
 }
}
]

This is due to the wonky MultipartUploadSlicingAlgorithm slicing into (31) 32 MB parts then creating a remainder 64 MB part that BaseBlobStore.putMultipartBlob does not account for. This interface seems stupid and better to create (32) 33 MB parts. Originally reported at:

https://stackoverflow.com/questions/47048264/the-number-of-source-components-provided-78-exceeds-the-maximum-32

Issue Links

• relates to: JCLOUDS-1606 Cannot upload more than 32 parts to GCS
• relates to: JCLOUDS-1369 MultipartUploadSlicingAlgorithm makes suboptimal choices with few part sizes