[HDDS-9229] [FSO] S3A compatibility - dfs -put creates dir and a file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Ashish Kumar
• Reporter: Mladjan Gadzic
• Created: 30/Aug/23 15:02
• Updated: 26/Feb/24 10:14
• Resolved: 02/Nov/23 01:58

Description

Bucket with FSO layout (i.e. /s3v/fso) is created on Ozone side (unsecured Ozone Docker cluster).
On Hadoop side (official 3.3.4 release) hdfs dfs -put command is issued (i.e README.txt file to be put to s3a://fso/).

What happens is: 2 keys are created (dir /fso/README.txt/ and file README.txt/README.txt.COPYING)

Expected behavior: 1 key is created, file /fso/README.txt
 
Logs from Ozone:

bash-4.2$ ozone sh key list /s3v/fso
[ {
  "volumeName" : "s3v",
  "bucketName" : "fso",
  "name" : "README.txt/",
  "dataSize" : 0,
  "creationTime" : "2023-08-30T14:56:49.675Z",
  "modificationTime" : "2023-08-30T14:56:49.675Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },
  "metadata" : { },
  "file" : false
}, {
  "volumeName" : "s3v",
  "bucketName" : "fso",
  "name" : "README.txt/EADME.txt._COPYING_",
  "dataSize" : 175,
  "creationTime" : "2023-08-30T14:56:49.675Z",
  "modificationTime" : "2023-08-30T14:56:50.295Z",
  "replicationConfig" : {
    "replicationFactor" : "ONE",
    "requiredNodes" : 1,
    "replicationType" : "RATIS"
  },
  "metadata" : { },
  "file" : true
} ]

 
Logs from Hadoop:

bin/hdfs dfs -Dfs.s3a.access.key=1 -Dfs.s3a.secret.key=1 -Dfs.s3a.endpoint=http://localhost:9878 -Dfs.s3a.path.style.access=true -put README.txt s3a://fso/
2023-08-30 14:56:41,596 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2023-08-30 14:56:41,747 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-08-30 14:56:41,747 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
2023-08-30 14:56:43,663 INFO impl.DirectoryPolicyImpl: Directory markers will be kept
2023-08-30 14:56:50,503 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2023-08-30 14:56:50,503 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2023-08-30 14:56:50,504 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.

Hadoop configuration (core-site.xml)

 

<configuration>
<property>
  <name>fs.s3a.access.key</name>
  <value>1</value>
</property>
<property>
  <name>fs.s3a.secret.key</name>
  <value>1</value>
</property><property>
  <name>fs.s3a.capability.directory.marker.policy.delete</name>
  <value>keep</value>
</property>
<property>
  <name>fs.s3a.capability.directory.marker.action.delete</name>
  <value>keep</value>
</property>
<property>
  <name>fs.s3a.directory.marker.retention</name>
  <value>keep</value>
</property>
</configuration> 

 

Issue Links

• relates to: HDDS-8423 S3 API compatibility improvements and fixes
• links to: GitHub Pull Request #5422