Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1368

GCS cannot upload multipart file larger than 1 GB

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      When uploading a file:

      long size = 33L * (32L * 1024L * 1024L);
      ByteSource byteSource = TestUtils.randomByteSource().slice(0, size);
      blobStore.createContainerInLocation(null, containerName);
      Blob blob = blobStore.blobBuilder(blobName)
            .payload(byteSource)
            .contentLength(size)
            .build();
      blobStore.putBlob(containerName, blob, PutOptions.Builder.multipart());
      

      I see an error:

      org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/gaul-blobstore0/o/large-blob/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
       "error": {
        "errors": [
         {
          "domain": "global",
          "reason": "invalid",
          "message": "The number of source components provided (33) exceeds the maximum (32)"
         }
        ],
        "code": 400,
        "message": "The number of source components provided (33) exceeds the maximum (32)"
       }
      }
      ]
      

      This is due to the wonky MultipartUploadSlicingAlgorithm slicing into (31) 32 MB parts then creating a remainder 64 MB part that BaseBlobStore.putMultipartBlob does not account for. This interface seems stupid and better to create (32) 33 MB parts. Originally reported at:

      https://stackoverflow.com/questions/47048264/the-number-of-source-components-provided-78-exceeds-the-maximum-32

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gaul Andrew Gaul
            gaul Andrew Gaul
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment