Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14627

Support MSI and DeviceCode token provider in ADLS

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
    • Component/s: fs/adl
    • Labels:
      None
    • Environment:

      MSI Change applies only to Hadoop running in an Azure VM

      Description

      This change is to upgrade the Hadoop ADLS connector to enable new auth features exposed by the ADLS Java SDK.

      Specifically:
      MSI Tokens: MSI (Managed Service Identity) is a way to provide an identity to an Azure Service. In the case of VMs, they can be used to give an identity to a VM deployment. This simplifies managing Service Principals, since the creds don’t have to be managed in core-site files anymore. The way this works is that during VM deployment, the ARM (Azure Resource Manager) template needs to be modified to enable MSI. Once deployed, the MSI extension runs a service on the VM that exposes a token endpoint to http://localhost at a port specified in the template. The SDK has a new TokenProvider to fetch the token from this local endpoint. This change would expose that TokenProvider as an auth option.

      DeviceCode auth: This enables a token to be obtained from an interactive login. The user is given a URL and a token to use on the login screen. User can use the token to login from any device. Once the login is done, the token that is obtained is in the name of the user who logged in. Note that because of the interactive login involved, this is not very suitable for job scenarios, but can work for ad-hoc scenarios like running “hdfs dfs” commands.

        Attachments

        1. HADOOP-14627-001.patch
          10 kB
          Atul Sikaria
        2. HADOOP-14627.004.patch
          15 kB
          John Zhuge
        3. HADOOP-14627.003.patch
          15 kB
          John Zhuge
        4. HADOOP-14627.002.patch
          15 kB
          Atul Sikaria

          Issue Links

            Activity

              People

              • Assignee:
                ASikaria Atul Sikaria
                Reporter:
                ASikaria Atul Sikaria
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: