[FLINK-26087] S3 filesystem tests could be enabled to run on PR AzureCI builds as well

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.13.5, 1.14.3, 1.15.0
• Component/s: Connectors / FileSystem
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 11/Feb/22 08:42
• Updated: 03/Mar/22 12:54
• Resolved: 03/Mar/22 12:54

Description

When looking into FLINK-26015 I noticed that the FileSystemBehaviorTestSuite doesn't run because of missing S3 credentials. I checked it only for the HadoopS3FileSystemBehaviorITCase on master in this build, though:

2022-02-11T07:35:43.3424031Z Feb 11 07:35:43 [INFO] Running org.apache.flink.fs.s3hadoop.HadoopS3RecoverableWriterExceptionITCase
2022-02-11T07:35:43.3574806Z Feb 11 07:35:43 [INFO] Running org.apache.flink.fs.s3hadoop.HadoopS3FileSystemBehaviorITCase
2022-02-11T07:35:43.9597611Z Feb 11 07:35:43 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 s - in org.apache.flink.fs.s3hadoop.HadoopS3RecoverableWriterExceptionITCase
2022-02-11T07:35:43.9818566Z Feb 11 07:35:43 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.612 s - in org.apache.flink.fs.s3hadoop.HadoopS3FileSystemBehaviorITCase
2022-02-11T07:35:44.6487150Z Feb 11 07:35:44 [ERROR] Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2022-02-11T07:35:44.6953455Z Feb 11 07:35:44 [ERROR] Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2022-02-11T07:35:45.0832407Z Feb 11 07:35:45 [INFO] Running org.apache.flink.fs.s3hadoop.HadoopS3FileSystemITCase
2022-02-11T07:35:45.1279447Z Feb 11 07:35:45 [INFO] Running org.apache.flink.fs.s3hadoop.HadoopS3RecoverableWriterITCase
2022-02-11T07:35:45.7356037Z Feb 11 07:35:45 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.636 s - in org.apache.flink.fs.s3hadoop.HadoopS3FileSystemITCase
2022-02-11T07:35:45.8154233Z Feb 11 07:35:45 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.677 s - in org.apache.flink.fs.s3hadoop.HadoopS3RecoverableWriterITCase
2022-02-11T07:35:46.1720757Z Feb 11 07:35:46 [INFO] 
2022-02-11T07:35:46.1721277Z Feb 11 07:35:46 [INFO] Results:
2022-02-11T07:35:46.1721728Z Feb 11 07:35:46 [INFO] 
2022-02-11T07:35:46.1722528Z Feb 11 07:35:46 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

I'd assume that this holds true for other test where S3TestCredentials.assumeCredentialsAvailable() is used. A MinioTestContainer was introduced as part of FLINK-26015 that could be utilized to work around this issue.

Issue Links

• Discovered while testing: FLINK-26015 FileSystemJobResultStore fails to access Minio
• duplicates: FLINK-12521 Use minio as AWS S3 alternative in end-to-end tests
• is related to: FLINK-26061 FileSystem.getFileStatus fails for empty directories on Presto S3 FS