[JCLOUDS-1623] Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.0
• Component/s: jclouds-blobstore
• Labels: aws-s3
• Assignee: Andrew Gaul
• Reporter: Ivan Dimitrov
• Created: 20/Feb/23 11:29
• Updated: 22/Mar/24 14:29

Description

Hello,

We are observing an issue with JClouds while transferring a file with an input stream only.
We are opening an input stream to a remote file and we are passing the stream to the JClouds library, but after some time it fails with the following error:

Exception in thread "main" java.lang.RuntimeException: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
	at com.google.common.base.Throwables.propagate(Throwables.java:241)
	at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
	at org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
	at org.example.Main.main(Main.java:50)
Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
	at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
	at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
	... 5 more

Note: The issue occurs only for larger files (more than 32 mb).
Java version: sapmachine-jdk-11.0.15.0.1.jdk
IAAS: AWS-s3

If we pass a BufferedInputStream the code works, so I guess that there is some issue with the processing of the stream. The stream returned by the Java Http Client is HttpResponseInputStream.

Code:

Main.java

package org.example;

import com.google.common.collect.ImmutableSet;
import com.google.inject.Module;
import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.PutOptions;
import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;

import java.io.BufferedInputStream;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.UUID;

public class Main {

    public static void main(String[] args) throws Exception {
        Iterable<Module> modules = ImmutableSet.<Module>of(
                new SLF4JLoggingModule());
        ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
                .credentials("<IDENTITY>", "<CREDENTIAL>")
                .modules(modules);
        BlobStoreContext blobStoreContext = contextBuilder.buildView(BlobStoreContext.class);
        BlobStore blobStore = blobStoreContext.getBlobStore();
        String decodedUrl = "<URL>";
        HttpClient client = buildHttpClient(decodedUrl);
        HttpResponse<InputStream> response = callRemoteEndpointWithRetry(client, decodedUrl);
        long fileSize = response.headers()
                .firstValueAsLong("Content-Length")
                .orElseThrow(() -> new IllegalArgumentException("No Content-Length"));
        System.out.println("Bytes: " + fileSize);
        InputStream inputStream = response.body();

            String container = "<CONTAINER>";
            Blob blob = blobStore.blobBuilder("TEST.zip")
//                    .payload(new BufferedInputStream(inputStream, 8 * 1024))
                    .payload(inputStream)
                    .contentDisposition(UUID.randomUUID().toString())
                    .contentType("application/octet-stream")
                    .contentLength(fileSize)
                    .build();
            blobStore.putBlob(container, blob, PutOptions.Builder.multipart());
    }

    private static HttpClient buildHttpClient(String decodedUrl) {
        return HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_1_1)
                .connectTimeout(Duration.ofMinutes(60))
                .followRedirects(HttpClient.Redirect.NORMAL)
                .authenticator(buildPasswordAuthenticator(decodedUrl))
                .build();
    }

    private static Authenticator buildPasswordAuthenticator(String decodedUrl) {
        return new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                var uri = URI.create(decodedUrl);
                var userInfo = uri.getUserInfo();
                if (userInfo != null) {
                    var separatorIndex = userInfo.indexOf(':');
                    var username = userInfo.substring(0, separatorIndex);
                    var password = userInfo.substring(separatorIndex + 1);
                    return new PasswordAuthentication(username, password.toCharArray());
                }
                return super.getPasswordAuthentication();
            }
        };
    }

    private static HttpResponse<InputStream> callRemoteEndpointWithRetry(HttpClient client, String decodedUrl) throws Exception {
        var response = client.send(buildFetchFileRequest(decodedUrl), HttpResponse.BodyHandlers.ofInputStream());
        if (response.statusCode() / 100 != 2) {
            throw new IllegalStateException("INVALID CODE " + response.statusCode());
        }
        return response;
    }

    private static HttpRequest buildFetchFileRequest(String decodedUrl) {
        var builder = HttpRequest.newBuilder()
                .GET()
                .expectContinue(true)
                .headers("Content-Type", "multipart/form-data")
                .timeout(Duration.ofMinutes(30));
        var uri = URI.create(decodedUrl);
        var userInfo = uri.getUserInfo();
        if (userInfo != null) {
            builder.uri(URI.create(decodedUrl.replace(userInfo + "@", "")));
        } else {
            builder.uri(uri);
        }
        return builder.build();
    }

}

Could you please take a look? Any advice would be helpful. The one thing I see differences between the HttpResponseInputStream and the BufferedInputStream is that HttpResponseInputStream::available returns 0. But I'm not sure if that's the issue because the skip of the stream is done by just reading it.

Note: Removed the binary content from jclouds-wire.log

Thanks!