[HDDS-5975] S3G: object key name should be encoded

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.3.0
• Component/s: S3
• Labels: pull-request-available
• Assignee: Kohei Sugihara
• Reporter: Kohei Sugihara
• Created: 12/Nov/21 06:09
• Updated: 22/Dec/21 20:23
• Resolved: 22/Dec/21 20:23

Description

S3G will not work properly for some objects that is includes `+` and ` ` in a key/prefix name. They should be url-encoded.

Quick reproduce

`aws s3 rm --recursive` will not work correctly because it confuse `+` and ` `.

# create files and sync them
% ls special-files -l
total 0
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:39 '&=;:+ ,?'
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:36 '<abc>'
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:41 '\{^}%'
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:41 '\{^}%`]>[~<#|'
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:24 'a b c'
-rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:21  a+b+c

% aws s3 --endpoint HOST sync special-files s3://should-be-deleted
upload: special-files/&=;:+ ,? to s3://should-be-deleted/&=;:+ ,?
upload: special-files/<abc> to s3://should-be-deleted/<abc>
upload: special-files/a b c to s3://should-be-deleted/a b c
upload: special-files/a+b+c to s3://should-be-deleted/a+b+c
upload: special-files/\{^}%`]>[~<#| to s3://should-be-deleted/\{^}%`]>[~<#|
upload: special-files/\{^}% to s3://should-be-deleted/\{^}%

# two 'a b c' files in this bucket. '+' and ' ' should be identified
% aws s3 --endpoint HOST ls s3://should-be-deleted/
2021-11-12 14:34:29          0 &=;:  ,?
2021-11-12 14:34:29          0 <abc>
2021-11-12 14:34:29          0 \{^}%
2021-11-12 14:34:29          0 \{^}%`]>[~<#|
2021-11-12 14:34:29          0 a b c
2021-11-12 14:34:29          0 a b c

# let's delete all files in the bucket
% aws s3 --endpoint HOST rm --recursive s3://should-be-deleted/
delete: s3://should-be-deleted/&=;:  ,?
delete: s3://should-be-deleted/<abc>
delete: s3://should-be-deleted/a b c
delete: s3://should-be-deleted/\{^}%
delete: s3://should-be-deleted/a b c
delete: s3://should-be-deleted/\{^}%`]>[~<#|

# several objects still remain
% aws s3 --endpoint HOST ls s3://should-be-deleted
2021-11-12 14:34:29          0 &=;:  ,?
2021-11-12 14:34:29          0 a b c

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>should-be-deleted</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <KeyCount>2</KeyCount>
  <Delimiter>/</Delimiter>
  <EncodingType>url</EncodingType>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>&amp;=;:+ ,?</Key>
    <LastModified>2021-11-12T05:34:29.337Z</LastModified>
    <ETag>2021-11-12T05:34:29.337Z</ETag>
    <Size>0</Size>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>a+b+c</Key>
    <LastModified>2021-11-12T05:34:29.610Z</LastModified>
    <ETag>2021-11-12T05:34:29.610Z</ETag>
    <Size>0</Size>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

References

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html – a guideline for object key name & special handlings
• https://issues.apache.org/jira/browse/HDDS-3161 – plus (`+`) is allowed here 

 

Issue Links

• links to: GitHub Pull Request #2834