Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1478

Error with multipart upload when blob uses byteArrayInputStream as payload

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.1
    • Fix Version/s: None
    • Component/s: jclouds-blobstore
    • Labels:
      None

      Description

      Hello,

      When trying to multipart upload 100MB file for example, and set ByteArrayInputStream as payload for the blob it fails with: Caused by: java.io.EOFException: reached end of stream after skipping 1198905 bytes; 100663296 bytes expected

      Here is the error trace:

      Dec 19, 2018 11:09:17 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3Client.public abstract java.lang.String org.jclouds.s3.S3Client.uploadPart(java.lang.String,java.lang.String,int,java.lang.String,org.jclouds.io.Payload)[xxx.xxx.xxx.bucket, C:\xxx\xxx\xxx\xxx\xxx.zip, 2, ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr, [content=true, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=33554432, contentMD5=null, contentType=application/unknown, expires=null], written=false, isSensitive=false]], request=PUT https://xxx.xxx.xxx.bucket.s3-eu-central-1.amazonaws.com/C:%5Cxxx%5Cxxxv%5Cxxx%5Cxxx%5Cxxx.zip?partNumber=2&uploadId=ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr HTTP/1.1]
      Dec 19, 2018 11:09:17 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3Client.public abstract java.lang.String org.jclouds.s3.S3Client.uploadPart(java.lang.String,java.lang.String,int,java.lang.String,org.jclouds.io.Payload)[xxx.xxx.xxx.bucket, C:\xxx\xxx\xxx\xxx\xxx.zip, 1, ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr, [content=true, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=33554432, contentMD5=null, contentType=application/unknown, expires=null], written=false, isSensitive=false]], request=PUT https://xxx.xxx.xxx.bucket.s3-eu-central-1.amazonaws.com/C:%5Cxxx%5Cxxx%5Cxxx%5xxx%5Cxx.zip?partNumber=1&uploadId=ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr HTTP/1.1]
      Dec 19, 2018 11:09:17 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3Client.public abstract java.lang.String org.jclouds.s3.S3Client.uploadPart(java.lang.String,java.lang.String,int,java.lang.String,org.jclouds.io.Payload)[xxx.xxx.xxx.bucket, C:\xxx\xxx\xxx\xxx\xxx.zip, 3, ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr, [content=true, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=33554432, contentMD5=null, contentType=application/unknown, expires=null], written=false, isSensitive=false]], request=PUT https://xxx.xxx.xxx.bucket.s3-eu-central-1.amazonaws.com/C:%5Cxxx%5Cxxx%5Cxxx%5Clxxx%5Cxxx.zip?partNumber=3&uploadId=ycoF6GcM8sRKx2jFfdzcHnn.YSRv8ewWzrIgqPI7f85B8fcQqcnqTwaqQKajRjNwnXVNJ3VBbQ8HtB6lP8aLFMLok7kIrbJ4Ir1ipxQH.fkxNVPuTi.445aHG_aOPQQr HTTP/1.1]
      Exception in thread "main" java.lang.RuntimeException: java.io.EOFException: reached end of stream after skipping 1198905 bytes; 100663296 bytes expected
       at com.google.common.base.Throwables.propagate(Throwables.java:160)
       at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
       at org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
       at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:371)
       at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:347)
       at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
       at Test.uploadMultipart(Test.java:47)
       at Test.main(Test.java:140)
      Caused by: java.io.EOFException: reached end of stream after skipping 1198905 bytes; 100663296 bytes expected
       at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:668)
       at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
       ... 6 more
      

       
      Here is the code I am using:

      private static S3Client mS3Client;

      private static ContextBuilder getJCloudContextBuilder() {
      return ContextBuilder.newBuilder("aws-s3").credentials(ACCESS_KEY, SECRET_KEY);
      }

      private static File getFile() {
      return new File(ONE_HUNDRED_MB_FILENAME);
      }

      private static void uploadMultipart() {
      BlobStoreContext blobStoreContext = getJCloudContextBuilder().build(BlobStoreContext.class);
      mS3Client = blobStoreContext.unwrapApi(S3Client.class);
      mS3Client.bucketExists(BUCKET_NAME);
      BlobStore blobStore = blobStoreContext.getBlobStore();
      try (InputStream inputStream = new ByteArrayInputStream(Files.readAllBytes(getFile().toPath()))){
      Blob blob = blobStore.blobBuilder(ONE_HUNDRED_MB_FILENAME).payload(inputStream).contentLength(getFile().length()).build();
      blobStore.putBlob(BUCKET_NAME, blob, PutOptions.Builder.multipart());
      } catch (FileNotFoundException e)

      { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); }

      }

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              blagolaj Blagoi Anastasov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: