[JCLOUDS-1389] Multipart upload to Google cloud storage failing with 400 bad request

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0, 2.1.1
• Component/s: jclouds-blobstore
• Labels: google-cloud-storage
• Environment: Windows 10, JDK 1.8.0_144, Jclouds 2.1.0
• Assignee: Andrew Gaul
• Reporter: Chris Bailey
• Created: 05/Mar/18 19:27
• Updated: 21/Apr/18 20:29
• Resolved: 21/Apr/18 20:29

Description

I am encountering an issue with multipart uploads.  What I am seeing is whenever I do a multipart upload, I get a "400 Bad Request" response. Here is the code I am using to upload:

String fileContents = Files.toString(new File("google_creds.json"), Charset.defaultCharset());

Supplier<Credentials> credentialSupplier = new GoogleCredentialsFromJson(fileContents);

Iterable<Module> modules = ImmutableSet.<Module>of(
new SLF4JLoggingModule());

BlobStoreContext context = ContextBuilder.newBuilder("google-cloud-storage")
.credentialsSupplier(credentialSupplier)
.modules(modules)
.buildView(BlobStoreContext.class);

BlobStore client = context.getBlobStore();

byte[] testBytes = new byte[102400];

Blob blob = client.blobBuilder("testupload").payload(testBytes)
.contentLength(testBytes.length).build();

// Multipart upload fails
client.putBlob("filecatalyst-unit-tests", blob, multipart());

Here is the HTTP request/response:

>> "{"sourceObjects":[{"name":"3fda0043-e340-4ccb-a582-9e3b78cf85e1_00000001","generation":1520026856033315,"objectPreconditions":{"ifGenerationMatch":1520026856033315}}],"destination":{"name":"testupload","size":102400,"contentType":"application/unknown","storageClass":"STANDARD","metadata":"}}
>> POST https://www.googleapis.com/storage/v1/b/filecatalyst-unit-tests/o/testupload/compose HTTP/1.1
>> Accept: application/json
>> Authorization: Bearer ***********************
>> Content-Type: application/json
>> Content-Length: 293
<< HTTP/1.1 400 Bad Request
<< Alt-Svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
<< Server: UploadServer
<< X-GUploader-UploadID: AEnB2UpqKslm87V4wto81YsNphtPFtybHjWgffGAC3XPE-xQAXSjDldst5s5nuWPyhXjEGOwWE5kDETjNA1EedbuEw9Vbe8CtA
<< Vary: X-Origin
<< Vary: Origin
<< Date: Fri, 02 Mar 2018 21:40:56 GMT
<< Content-Type: application/json; charset=UTF-8
<< Content-Length: 178
<< "{[\n]"
<< " "error": {[\n]"
<< " "errors": [[\n]"
<< " {[\n]"
<< " "domain": "global",[\n]"
<< " "reason": "invalid",[\n]"
<< " "message": "Invalid argument"[\n]"
<< " }[\n]"
<< " ],[\n]"
<< " "code": 400,[\n]"
<< " "message": "Invalid argument"[\n]"
<< " }[\n]"
<< "}[\n]"

Here is the formatted JSON request body for ease of reading:

{
  "sourceObjects":[
    {
      "name":"83116375-0ba1-4d1a-aec3-8dd29dab0da9_00000001",
      "generation":1520263556434885,
      "objectPreconditions":

Unknown macro: {         "ifGenerationMatch"}

    }
  ],
  "destination":{
    "name":"testupload",
    "size":33554432,
    "contentType":"application/unknown",
    "storageClass":"STANDARD",
    "metadata":{}
  }
}

 

The issue is reproducible with various file sizes, have tried up to 1 GB.