[HADOOP-16567] S3A Secret access to fall back to XML if credential provider raises IOE.

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 12/Sep/19 14:56
• Updated: 04/Oct/22 16:23

Description

This is hive related. Hive can put secrets into a JCEKS file which only hive may read.

S3A file systems created on behalf of a user do not have access to this file. Yet it is listed as the credential provider in the hadoop.credential.providers option in core-site -and that is marked as final. When the S3 a initializre() method looks up passwords and encryption keys, the failure to open the file raises an IOE -and the FS cannot be instantiated.

Proposed: S3AUtils.lookupPassword() to catch such exceptions, and fall back to using Configuration.get() and so retrieve any property in the XML. If there is one failing here, it is if the user did want to read from a credential provider, the failure to read the credential will be lost, and the filesystem will simply get the default value.

There is a side issue, that permission exceptions can surface as found not found exceptions, which are then wrapped as generic IOEs in Configuration. It will be hard and brittle to attempt to only respond to permission restrictions. We could look at improving Configuration.getPassword() but that class is so widely used, I am not in a rush to break things.

I think this means we have to add another option. Trying to be clever about when to fall back versus when to rethrow the exception is doomed.

If this works for S3A, we will need to consider replicating it for ABFS.

Issue Links

• is duplicated by: HADOOP-16607 s3a attempts to look up password/encryption fail if JCEKS file unreadable