[JCLOUDS-1606] Cannot upload more than 32 parts to GCS

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.6.0
• Component/s: jclouds-blobstore
• Labels: google-cloud-storage
• Assignee: Andrew Gaul
• Reporter: Lari Hotari
• Created: 13/Apr/22 07:03
• Updated: 26/Feb/24 05:41
• Resolved: 26/Feb/24 05:41

Description

There's currently a limitation in JClouds that it cannot upload more than 32 parts to GCS.

org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
  "error": {
    "code": 400,
    "message": "The number of source components provided (35) exceeds the maximum (32)",
    "errors": [
      {
        "message": "The number of source components provided (35) exceeds the maximum (32)",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
} 

The limitation of 32 parts is per API call to the compose endpoint.

 

When there are more than 32 parts, the endpoint should be called multiple times. The total limit is 10000 parts in GCS.

https://cloud.google.com/storage/docs/composite-objects

 

When you perform a composition:

• The source objects are unaffected.
• You can use between 1 and 32 source objects.
• Source objects can themselves be composite objects.

 

Issue Links

• is related to: JCLOUDS-1368 GCS cannot upload multipart file larger than 1 GB
• is related to: JCLOUDS-866 Investigate GCS multipart upload limits
• requires: JCLOUDS-1608 Slicing of large files can lead to exceed the 32 parts limit of GCS