[HDDS-7594] [FSO] Folders created through S3G are created on file system as "files".

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Mladjan Gadzic
• Reporter: Neil Joshi
• Created: 05/Dec/22 23:29
• Updated: 27/Mar/24 21:31
• Resolved: 14/Feb/23 20:02

Description

With FSO bucket layout buckets, folders created with the s3-gateway create 0 byte files on file system.  Unlike Object Store bucket layout buckets that create folders correctly when created through the s3-gateway, folder creation on FSO buckets with s3 gateway results in 0 byte files.  Expected is the same behaviour.  This has caused problems with interoperability for applications accessing ozone through both ofs and the s3 gateway, such as Trino.

 

Problem can be produced issuing aws-cli folder creation to the s3 gateway on an ozone cluster layout FILE_SYSTEM_OPTIMIZED:

 

bash-4.2$ ozone sh bucket create --layout=FILE_SYSTEM_OPTIMIZED /s3v/bucketfso
ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object --bucket bucketfso --key test1
ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object --bucket bucketfso --key test1/README.md --body ./README.md
An error occurred (InvalidRequest) when calling the PutObject operation: An error occurred (InvalidRequest) when calling the PutObject/MPU PartUpload operation: ozone.om.enable.filesystem.paths is enabled Keys are considered as Unix Paths. Path has Violated FS Semantics which caused put operation to fail.

bash-4.2$ ozone fs -ls ofs://om/s3v/bucketfso Found 2 items -rw-rw-rw-   1 hadoop hadoop          0 2022-12-05 22:34 ofs://om/s3v/bucketfso/test1

File is created when a folder is expected.  If it were correctly set it would result in seeing drwxrwxrwx for the test1 key as in,

drwxrwxrwx   - hadoop hadoop          0 2022-12-05 22:40 ofs://om/s3v/bucketfos/test1

 

In contrast, the  same aws command issued on a OBJECT_STORE layout bucket results in :

bash-4.2$ ozone sh key list /s3v/bucketobs
[ {
  "volumeName" : "s3v",
  "bucketName" : "bucketobs",
  "name" : "test1",
  "dataSize" : 0,
  "creationTime" : "2022-12-05T22:33:21.737Z",
  "modificationTime" : "2022-12-05T22:33:21.746Z",
  "replicationConfig" : {
    "replicationFactor" : "ONE",
    "requiredNodes" : 1,
    "replicationType" : "RATIS"
  },
  "metadata" : { }
}, {
  "volumeName" : "s3v",
  "bucketName" : "bucketobs",
  "name" : "test1/README.md",
  "dataSize" : 3879,
  "creationTime" : "2022-12-05T22:34:02.402Z",
  "modificationTime" : "2022-12-05T22:37:00.758Z",
  "replicationConfig" : {
    "replicationFactor" : "ONE",
    "requiredNodes" : 1,
    "replicationType" : "RATIS"
  },
  "metadata" : { }
} ]

As is expected.

Issue Links

• is related to: HDDS-7824 Document bucket layout related config keys
• relates to: HDDS-10570 dfs -touch creates directory instead of empty file in FSO bucket
• links to: GitHub Pull Request #4186