Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5975

S3G: object key name should be encoded

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.3.0
    • S3

    Description

      S3G will not work properly for some objects that is includes `+` and ` ` in a key/prefix name. They should be url-encoded.

      Quick reproduce

      `aws s3 rm --recursive` will not work correctly because it confuse `+` and ` `.

      # create files and sync them
      % ls special-files -l
      total 0
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:39 '&=;:+ ,?'
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:36 '<abc>'
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:41 '\{^}%'
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:41 '\{^}%`]>[~<#|'
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:24 'a b c'
      -rw-r--r-- 1 ksugihara ksugihara 0 Nov 12 08:21  a+b+c
      
      % aws s3 --endpoint HOST sync special-files s3://should-be-deleted
      upload: special-files/&=;:+ ,? to s3://should-be-deleted/&=;:+ ,?
      upload: special-files/<abc> to s3://should-be-deleted/<abc>
      upload: special-files/a b c to s3://should-be-deleted/a b c
      upload: special-files/a+b+c to s3://should-be-deleted/a+b+c
      upload: special-files/\{^}%`]>[~<#| to s3://should-be-deleted/\{^}%`]>[~<#|
      upload: special-files/\{^}% to s3://should-be-deleted/\{^}%
      
      # two 'a b c' files in this bucket. '+' and ' ' should be identified
      % aws s3 --endpoint HOST ls s3://should-be-deleted/
      2021-11-12 14:34:29          0 &=;:  ,?
      2021-11-12 14:34:29          0 <abc>
      2021-11-12 14:34:29          0 \{^}%
      2021-11-12 14:34:29          0 \{^}%`]>[~<#|
      2021-11-12 14:34:29          0 a b c
      2021-11-12 14:34:29          0 a b c
      
      # let's delete all files in the bucket
      % aws s3 --endpoint HOST rm --recursive s3://should-be-deleted/
      delete: s3://should-be-deleted/&=;:  ,?
      delete: s3://should-be-deleted/<abc>
      delete: s3://should-be-deleted/a b c
      delete: s3://should-be-deleted/\{^}%
      delete: s3://should-be-deleted/a b c
      delete: s3://should-be-deleted/\{^}%`]>[~<#|
      
      # several objects still remain
      % aws s3 --endpoint HOST ls s3://should-be-deleted
      2021-11-12 14:34:29          0 &=;:  ,?
      2021-11-12 14:34:29          0 a b c
      
      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
      <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <Name>should-be-deleted</Name>
        <Prefix></Prefix>
        <Marker></Marker>
        <MaxKeys>1000</MaxKeys>
        <KeyCount>2</KeyCount>
        <Delimiter>/</Delimiter>
        <EncodingType>url</EncodingType>
        <IsTruncated>false</IsTruncated>
        <Contents>
          <Key>&amp;=;:+ ,?</Key>
          <LastModified>2021-11-12T05:34:29.337Z</LastModified>
          <ETag>2021-11-12T05:34:29.337Z</ETag>
          <Size>0</Size>
          <StorageClass>STANDARD</StorageClass>
        </Contents>
        <Contents>
          <Key>a+b+c</Key>
          <LastModified>2021-11-12T05:34:29.610Z</LastModified>
          <ETag>2021-11-12T05:34:29.610Z</ETag>
          <Size>0</Size>
          <StorageClass>STANDARD</StorageClass>
        </Contents>
      </ListBucketResult>
      

      References

       

      Attachments

        Issue Links

          Activity

            People

              ksugihara Kohei Sugihara
              ksugihara Kohei Sugihara
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: