[HDDS-9762] [FSO] Hadoop dfs s3a protocol does not work with FSO buckets

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Ashish Kumar
• Reporter: Mladjan Gadzic
• Created: 22/Nov/23 18:51
• Updated: 26/Feb/24 10:19
• Resolved: 07/Dec/23 13:24

Description

Trying to exercise freon dfsg over s3a results in exception.

Command:

 

OZONE_CLASSPATH=/opt/hadoop/share/ozone/lib/aws-java-sdk-bundle-1.11.1026.jar:/opt/hadoop/share/ozone/lib/hadoop-aws-3.3.2.jar:$(ozone classpath ozone-common) ozone freon \-Dfs.s3a.endpoint=http://host.docker.internal:9878 \-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true \-Dfs.s3a.change.detection.source=versionid \-Dfs.s3a.change.detection.mode=client \-Dfs.s3a.change.detection.version.required=false \dfsg -s102400 -n10000 -t10 --path=s3a://fso/ --prefix="s3-1GB" 

 

Exception (command first run)

2023-11-22 18:34:19,180 [s3a-transfer-fso-unbounded-pool4-t1] DEBUG impl.BulkDeleteRetryHandler: Retrying on error during bulk delete
:org.apache.hadoop.fs.s3a.AWSS3IOException: delete: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS:null: InternalError: s3-1GB/: Directory is not empty. Key:s3-1GB
: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request ID: DwT29rWRhtYS; Proxy: null)
        at org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:117)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:312)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:426)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2775)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3022)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3121)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3078)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:4498)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$finishedWrite$31(S3AFileSystem.java:4403)
        at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS
        at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2345)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
        ... 11 more

In consecutive run (command second run), there is a different exception

2023-11-22 18:39:36,543 [pool-2-thread-9] ERROR freon.BaseFreonGenerator: Error on executing task 7
:org.apache.hadoop.fs.FileAlreadyExistsException: s3a://fso/s3-1GB/7 is a directory
 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1690)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1078)
 at org.apache.hadoop.ozone.freon.HadoopFsGenerator.lambda$createFile$0(HadoopFsGenerator.java:112)
 at com.codahale.metrics.Timer.time(Timer.java:101)
 at org.apache.hadoop.ozone.freon.HadoopFsGenerator.createFile(HadoopFsGenerator.java:111)
 at org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
 at org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
 at org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:829) 

Ozone SHA f34d347af1f7b9c1eb82cf27fbe8231c85493628.

Libs from Hadoop 3.3.2 version.

It is reproducible using unsecure Ozone Docker cluster with 3DNs.

Steps to reproduce the issue:

1. bring up unsecure Ozone Docker cluster
2. exec into OM container
3. add env variables 
   AWS_ACCESS_KEY_ID=random
   AWS_SECRET_KEY=random
   OZONE_ROOT_LOGGER=debug,console
4. create bucket named "fso" with FSO layout
5. run mentioned command (first time)
6. check output
7. run mentioned command (second time)
8. check output

Issue Links

• is related to: HDDS-8423 S3 API compatibility improvements and fixes
• relates to: HDDS-9825 ListKeys Iterator doesn't terminate when prefix contains "/" at the end
• links to: GitHub Pull Request #5735