[HADOOP-18857] AWS v2 SDK: fail meaningfully if legacy v2 signing requested

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: fs/s3
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 21/Aug/23 13:39
• Updated: 27/Jan/24 08:13
• Resolved: 12/Oct/23 16:49

Description

if the v2 signer is requested on an AWS SDKv2 build, you get an NPE.

proposed: special recognition of fs.s3a.signing-algorithm=S3SignerType and fail with a message that it isn't supported

> bin/hadoop jar $CLOUDSTORE storediag -D fs.s3a.signing-algorithm=S3SignerType -w  s3a://stevel-london/

2023-08-21 14:36:41,229 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for s3a://stevel-london/: 0:00:357
java.lang.NullPointerException
        at org.apache.hadoop.fs.s3a.auth.SignerFactory.createSigner(SignerFactory.java:104)
        at org.apache.hadoop.fs.s3a.impl.AWSClientConfig.createClientConfigBuilder(AWSClientConfig.java:90)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createClientOverrideConfiguration(DefaultS3ClientFactory.java:163)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:147)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:89)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:989)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:610)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)

Issue Links

• Is contained by: HADOOP-18889 S3A: V2 SDK client does not work with third-party store
• is related to: HADOOP-18747 AWS SDK V2 - sigv2 support