Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.1, 2.5.0
-
Patch
Description
MultipartUploadSlicingAlgorithm calculates slices for a large file by first using the defaultPartSize. If the results of that slicing gives parts that don't exceed the min/maxPartSizes (5MB and 5GB for GoogleCloudStorageBlobStore) but that do exceed the maxNumberOfParts (32 for GoogleCloudStorageBlobStore), the algoritm sets the number of parts to 32 and recalculates the size of the parts. If there is any remainder after that, JClouds ends up uploading 33 parts in total to GCS, causing the process to fail in completeMultipartUpload() when recomposing the original content from the parts.
The following simple unitTest proves the case:
public class AlgoritmTest extends TestCase {
public void testSlicing() {
MultipartUploadSlicingAlgorithm algorithm = new MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);
algorithm.calculateChunkSize(1024*1024*1200+33);
assertTrue(algorithm.getParts()+((algorithm.getRemaining() > 0)?(1):(0)) <= 32);
}
It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a remainder).
The following patch fixes the issue:
...
long remainder = length % unitPartSize;
// SHB patch
// remainder should be distributed over parts if we are at the maximumNumberOfParts
// (if not, an additional part is uploaded to GCS thus exceeding the maximum allowed parts)
// if (remainder == 0 && parts > 0) {
// parts -= 1;
if (remainder > 0 && parts == maximumNumberOfParts) {
parts -= 1;
partSize = length/parts;
// end of SHB patch
...
I also commented the code that reduces the number of parts when there is no remainder, since that ends up creating a remaining part that is the same size as the others.
Attachments
Issue Links
- is required by
-
-
- Closed
-