[HADOOP-18562] S3A: support custom S3 and STS headers

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.9
• Assignee: Prerak Pradhan
• Reporter: Daniel Carl Jones
• Created: 07/Dec/22 18:29
• Updated: 03/Dec/24 13:39

Description

Some users have the use case where for a particular S3A filesystem, they would like to set one or more headers to a static value.

We might imagine a set of configurations properties with a common prefix, where the suffix determines the header name and the value of the property determines the value of the header.

Per-filesystem headers can be set using the per-bucket configuration.

Issue Links

• is duplicated by: SPARK-38958 Override S3 Client in Spark Write/Read calls
• links to: GitHub Pull Request #6550