[FLINK-13602] S3 Presto filesystem effectively does not support credential providers - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Not a Priority
Resolution: Unresolved
Affects Version/s: 1.9.0
Fix Version/s: None
Component/s: FileSystems
Labels:
- auto-deprioritized-major
- auto-deprioritized-minor

Description

To provide credentials to S3 users may configure a credentials provider. For providers from amazon (which are relocated) we allow users to configure the original class name, and relocate it manually in the S3 filesystem factories.

However, none of the amazon provided credential providers can be used with the Presto filesystem, since it additionally requires them to have a constructor accepting a hadoop configuration. (https://prestodb.github.io/docs/current/connector/hive.html#amazon-s3-configuration)

hadoop-aws does include a number of credential providers that have this constructor, however these use configuration keys that aren't mirrored from the flink config. (they expect fs.s3a as a key-prefix), not to mention that users would have to configure the relocated class (since the S3 factory only manually relocates amazon classes).

Finally, a custom implementation of the credentials provider can effectively be ruled out since they too would have to be implemented against the relocated amazon/hadoop classes, which we can't really expect users to do.

In summary, amazon providers aren't working since they don't have a constructor that presto requires, hadoop providers don't work since we don't mirror the required configuration keys, and custom providers are unreasonable as they'd have to be implemented against relocated classes.