• Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.0
    • Fix Version/s: None
    • Component/s: security
    • Labels:
    • Target Version/s:


      Extend the idea from HADOOP-6520 "UGI should load tokens from the environment" to a generic lightweight "keychain" design. Load keys (secrets) into a keychain in UGI (secret map) at startup. YARN will distribute them securely into each container. The Hadoop code running in the container can then retrieve the credentials from UGI.

      The use case is Bring Your Own Key (BYOK) credentials for cloud connectors (adl, wasb, s3a, etc.), while Hadoop authentication is still Kerberos. No configuration change, no admin involved. It will support YARN applications initially, e.g., DistCp, Tera Suite, Spark-on-Yarn, etc.

      Implementation is surprisingly simple because almost all pieces are in place:

      • Retrieve secrets from UGI using conf.getPassword backed by the existing Credential Provider class UserProvider
      • Reuse Credential Provider classes and interface to define local permanent or transient credential store, e.g., LocalJavaKeyStoreProvider
      • New: create a new transient Credential Provider that logs into AAD with username/password or device code, and then put the Client ID and Refresh Token into the keychain
      • New: create a new permanent Credential Provider based on Hadoop configuration XML, for dev/testing purpose.



        1. HADOOP-14808.003.patch
          26 kB
          John Zhuge
        2. HADOOP-14808.002.patch
          26 kB
          John Zhuge
        3. HADOOP-14808.001.patch
          25 kB
          John Zhuge

          Issue Links



              • Assignee:
                jzhuge John Zhuge
                jzhuge John Zhuge
              • Votes:
                0 Vote for this issue
                14 Start watching this issue


                • Created: