Just saw this while working on SPARK-35878.
If you are copying endpoints then you may also want to think about picking up the region from AWS_REGION too.
The full list of env vars which I have collected by looking in the AWS SDKs is up at https://github.com/steveloughran/cloudstore/blob/main/src/main/java/org/apache/hadoop/fs/store/diag/S3ADiagnosticsInfo.java#L379
I do not know what they all mean or do, only that if I get a support call I want to know if anyone has been setting them.
Issue resolved by pull request 43205
https://github.com/apache/spark/pull/43205