Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16607

s3a attempts to look up password/encryption fail if JCEKS file unreadable



    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 3.2.1, 3.1.3
    • None
    • fs/s3, security
    • None


      Hive deployments can use a JCEKs file to store secrets, which it sets up
      To be readable only by the Hive user, listing it under hadoop.credential.providers.

      When it tries to create an S3A FS instance as another user, via a doAs{}
      clause, the S3A FS getPassword() call fails on the subsequent AccessDeniedException -even if the secret it is looking for is in the XML file
      or, as in the case of encryption settings, or session key undefined.

      I can you point the blame at hive for this -it's the one with a forbidden JCEKS file on the provider path, but I think it is easiest to fix in S3AUtils than
      in hive, and safer then changing Configuration.

      ABFS is likely to see the same problem.

      I propose an option to set the fallback policy.

      I initially thought about always handling this:
      Catching the exception, attempting to downgrade to Reading XML and if that fails rethrowing the caught exception.

      However, this will do the wrong thing if the option is completely undefined,
      As is common with the encryption settings.

      I don't want to simply default to log and continue here though, as it may be a legitimate failure -such as when you really do want to read secrets from such a source.

      Issue: what fallback policies?

      • fail: fail fast. today's policy; the default.
      • ignore: log and continue

      We could try and be clever in future. To get away with that, we would have to declare which options were considered compulsory and re-throw the caught
      Exception if no value was found in the XML file.

      That can be a future enhancement -but it is why I want the policy to be an enumeration, rather than a simple boolean.

      Tests: should be straightforward; set hadoop.credential.providers to a non-existent file and expected to be processed according to the settings.


        Issue Links



              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              0 Vote for this issue
              1 Start watching this issue