[HDDS-10521] ETag field should not be returned during GetObject if the key does not contain ETag field

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0, 1.4.1
• Component/s: S3, s3gateway
• Labels: pull-request-available
• Target Version/s: 1.5.0, 1.4.1
• Assignee: Ivan Andika
• Reporter: Ivan Andika
• Created: 14/Mar/24 02:57
• Updated: 25/Apr/24 01:41
• Resolved: 15/Mar/24 03:44

Description

A user encountered this error when it tries to download a key using the AWS S3 SDK (version 1.11.415).

ERROR FileOperationServiceImpl - s3Client.getObject invoke error, objectKey:<redacted>. (FileOperationServiceImpl.java:438)
java.lang.ArrayIndexOutOfBoundsException: 110
    at com.amazonaws.util.Base16Codec.pos(Base16Codec.java:96)
    at com.amazonaws.util.Base16Codec.decode(Base16Codec.java:87)
    at com.amazonaws.util.Base16Lower.decode(Base16Lower.java:53)
    at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:48)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1456)
    ...

Although the key was able to be downloaded using AWS s3api

aws s3api --endpoint <redacted> get-object --bucket <redacted> --key <redacted> download.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 28 Feb 2024 11:06:30 GMT",
    "ContentLength": 1117055,
    "ETag": "\"null\"",
    "CacheControl": "no-cache",
    "ContentType": "application/octet-stream",
    "Expires": "Wed, 13 Mar 2024 08:10:05 GMT",
    "Metadata": {}
} 

The problem can be replicated even with the current version by uploading a file with ofs and downloading it using AWS S3 SDK.

It seems object without ETag field was able to be downloaded using AWS CLI, but not AWS SDK. 

After looking at the AWS SDK code it seems that AWS SDK will do a post-processing step that will validate the ETag field of the downloaded object to the object's content. If the ETag field is null, the post-processing step will skip the validation.

Currently, S3G returns a string "null" for the ETag field if the ETag field does not exist, which should cause the AWS SDK to not be able to parse the string since it md5 string is longer than the "null" string. This is most probably why there is an ArrayIndexOutOfBoundsException

One possible solution is to not return the ETag field at all if the key does not contain ETag to begin with. This way the post processing step in the AWS SDK will not validate the md5 hash.

Future improvement: We might need to add some test coverage for AWS SDK (maybe in the integration tests) since the behavior might be different than AWS CLI.

Issue Links

• is caused by: HDDS-9115 Provide ETag in a response header on file downloading through s3g
• links to: GitHub Pull Request #6377