Details
Minor Description
Hive deployments can use a JCEKs file to store secrets, which it sets up
To be readable only by the Hive user, listing it under hadoop.credential.providers.
When it tries to create an S3A FS instance as another user, via a doAs{}
clause, the S3A FS getPassword() call fails on the subsequent AccessDeniedException -even if the secret it is looking for is in the XML file
or, as in the case of encryption settings, or session key undefined.
I can you point the blame at hive for this -it's the one with a forbidden JCEKS file on the provider path, but I think it is easiest to fix in S3AUtils than
in hive, and safer then changing Configuration.
ABFS is likely to see the same problem.
I propose an option to set the fallback policy.
I initially thought about always handling this:
Catching the exception, attempting to downgrade to Reading XML and if that fails rethrowing the caught exception.
However, this will do the wrong thing if the option is completely undefined,
As is common with the encryption settings.
I don't want to simply default to log and continue here though, as it may be a legitimate failure -such as when you really do want to read secrets from such a source.
Issue: what fallback policies?
- fail: fail fast. today's policy; the default.
- ignore: log and continue
We could try and be clever in future. To get away with that, we would have to declare which options were considered compulsory and re-throw the caught
Exception if no value was found in the XML file.
That can be a future enhancement -but it is why I want the policy to be an enumeration, rather than a simple boolean.
Tests: should be straightforward; set hadoop.credential.providers to a non-existent file and expected to be processed according to the settings.
Attachments
Issue Links
- duplicates
-
HADOOP-16567 S3A Secret access to fall back to XML if credential provider raises IOE.
-
- Open
-