[JCLOUDS-1505] BlobStore.blobMetadata(container, object) returns a StorageMetadata object with empty size when using org.jclouds.http.apachehc.config.ApacheHCHttpCommandExecutorServiceModule

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0, 2.1.3
• Component/s: jclouds-drivers
• Assignee: Andrew Gaul
• Reporter: Xavier BOURGOUIN
• Created: 31/Jul/19 16:04
• Updated: 10/Oct/19 17:02
• Resolved: 10/Oct/19 17:02

Description

Hello,

 

When calling blobStore.blobMetadata(container, objectName).getSize() with a blobStoreContext using ApacheHCHttpCommandExecutorServiceModule, the returned BlobMetadata object has a size attribute which is null.

Issue spotted while using jclouds 2.1.2 over an S3 object store (Scality, but seems like it is reproducible with any other S3 provider, and potentially others OS providers like GCS, to be confirmed).

In essence, the minimal reproducer pseudo-code is:

 BlobStoreContext context = ContextBuilder.newBuilder(provider)
         .credentials(identity, credential)
         .endpoint(endpoint)
         // usage of ApacheHCHttp module
         .modules(ImmutableList.<Module> of(new ApacheHCHttpCommandExecutorServiceModule(), new SLF4JLoggingModule()))
         .buildView(BlobStoreContext.class);
     
try {
  BlobStore blobStore = context.getBlobStore();
  // output is: "File size from metadata: null"
  System.out.println("File size from metadata: " + blobStore.blobMetadata(DUMMY_CONTAINER, DUMMY_FILE_NAME).getSize());
} finally {
  context.close();
}

 

For convenience, I've attached a ready to run reproducer to this ticket, here's how to run it:

 

> mkdir reproducer
> tar xvzf jclouds_null_size_reproducer.tar.gz -C reproducer
> cd reproducer
> mvn install
# note: pre-req from this stage is to have an S3 Object Store available, you can use minio one if you have docker (docker run -it -p 9000:9000 -v /mnt/data:/data minio/minio server /data)
# replace with your Object Store endpoint, identity and access key
> java -jar target/blobstore-basics-jar-with-dependencies.jar s3 http://10.66.33.1:9000 <identity here> <access key here>

File size from metadata: 8
File size from metadata (with ApacheHCHttpCommandExecutorServiceModule): null

The reproducer code has jclouds-wire logs enabled (inside log folder) and I've also added the log files inside the attached tarball for convenience. We see that in the HTTP HEAD response (the 2nd one, which correspond to my .blobMetadata() call using ApacheHCHttpCommandExecutorServiceModule), the Content-* headers are missing :

>cat log/jclouds-wire.log
# note: first .blobMetadata() call without ApacheHCHttpCommandExecutorServiceModule, the response contains Content-* headers
2019-07-31 17:30:13,257 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT
2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1"
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Vary: Origin
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B687995977CC49
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Type: application/unknown
2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Length: 8

# note: second call with ApacheHCHttpCommandExecutorServiceModule, the reponse is missing the content-* headers
2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT
2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1"
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Vary: Origin
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B6879961644613
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block
2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT

 

 I suspect the issue is coming from ApacheHCHttpCommandExecutorService.java#invoke method, it seems to be expecting the content-* headers are exclusively coming from the reponse.getEntity() (which I believe is null in case of a response to a HEAD request), because it strips it out of the headers list (which are well containing the content-* headers before the call to "filterOutContentHeaders"). A possible fix to cope with all requests kinds could be to strip it out only if it is already part of the payload (which I guess was the initial intent).

 

Environment details:

• jclouds version: 2.1.2 (but it is probably reproducible on older versions, to be confirmed)
• Operating system: Linux (kernel 4.15.0-55)
• Storage provider: S3 (reproduced on both Scality and MinIO), probably reproducible with other storage providers involving HTTP HEAD requests to get the metadata (GCS, ..)
• Java version:

  >java -version

• java version "1.8.0_201"
  Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
  Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)