[FLINK-24595] Programmatic configuration of S3 doesn't pass parameters to Hadoop FS

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Connectors / Hadoop Compatibility

Environment

Flink 1.14.0

JDK 8 

openjdk version "1.8.0_302"
OpenJDK Runtime Environment (Zulu 8.56.0.23-CA-macos-aarch64) (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (Zulu 8.56.0.23-CA-macos-aarch64) (build 25.302-b08, mixed mode)

• Assignee: Unassigned
• Reporter: Pavel Penkov
• Created: 19/Oct/21 17:36
• Updated: 10/May/23 03:06

Description

When running in mini-cluster mode Flink apparently doesn't pass S3 configuration to underlying Hadoop FS. With a code like this

Configuration conf = new Configuration();
conf.setString("s3.endpoint", "http://localhost:4566");
conf.setString("s3.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
conf.setString("s3.access.key", "harvester");
conf.setString("s3.secret.key", "harvester");
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(conf);

Application fails with an exception with most relevant error being Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Failed to connect to service endpoint: 

So Hadoop lists all the providers but it should use only the one set in configuration. Full project that reproduces this behaviour is available at https://github.com/PavelPenkov/flink-s3-conf and relevant files are attached to this issue.