Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: fs/adl
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Switch the default ADLS access token provider type from Custom to ClientCredential.

      Description

      Split off from a big patch in HADOOP-14038.

      Switch fs.adl.oauth2.access.token.provider.type default from Custom to ClientCredential and add ADLS properties to core-default.xml.

      Fix AdlFileSystem#getAccessTokenProvider which implies the provider type is Custom.
      Fix several unit tests that set dfs.adls.oauth2.access.token.provider but does not set dfs.adls.oauth2.access.token.provider.type.

        Activity

        Hide
        jzhuge John Zhuge added a comment -

        Atul Sikaria, Vishwajeet Dusane, Mingliang Liu, Chris Douglas, are you ok with default ClientCredential? Anybody for RefreshToken?

        Show
        jzhuge John Zhuge added a comment - Atul Sikaria , Vishwajeet Dusane , Mingliang Liu , Chris Douglas , are you ok with default ClientCredential ? Anybody for RefreshToken ?
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Yes ClientCredential as default should be fine. Atul Sikaria has updated document to specify type in HADOOP-14196

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Yes ClientCredential as default should be fine. Atul Sikaria has updated document to specify type in HADOOP-14196
        Hide
        jzhuge John Zhuge added a comment -

        Patch 001

        • Switch the default value for fs.adl.oauth2.access.token.provider.type from Custom to ClientCredential
        • Add config properties to core-default.xml
        Show
        jzhuge John Zhuge added a comment - Patch 001 Switch the default value for fs.adl.oauth2.access.token.provider.type from Custom to ClientCredential Add config properties to core-default.xml
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 12m 49s trunk passed
        +1 compile 16m 51s trunk passed
        +1 checkstyle 1m 53s trunk passed
        +1 mvnsite 1m 27s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 12s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 0m 58s the patch passed
        +1 compile 16m 5s the patch passed
        +1 javac 16m 5s the patch passed
        +1 checkstyle 2m 6s the patch passed
        +1 mvnsite 1m 42s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 22s the patch passed
        +1 javadoc 1m 18s the patch passed
        -1 unit 8m 4s hadoop-common in the patch failed.
        +1 unit 3m 43s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 40s The patch does not generate ASF License warnings.
        99m 19s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-14174
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860898/HADOOP-14174.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux cbda43cfcffc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6b09336
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 49s trunk passed +1 compile 16m 51s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 12s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 16m 5s the patch passed +1 javac 16m 5s the patch passed +1 checkstyle 2m 6s the patch passed +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 22s the patch passed +1 javadoc 1m 18s the patch passed -1 unit 8m 4s hadoop-common in the patch failed. +1 unit 3m 43s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 99m 19s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14174 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860898/HADOOP-14174.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux cbda43cfcffc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b09336 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11952/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        +1. It LGTM.

        It'd be nice to have the inputs from the rest of community. I will hold for 2 days before committing.

        Show
        eddyxu Lei (Eddy) Xu added a comment - +1. It LGTM. It'd be nice to have the inputs from the rest of community. I will hold for 2 days before committing.
        Hide
        jzhuge John Zhuge added a comment -

        I am committing it.

        Atul Sikaria, Vishwajeet Dusane, Mingliang Liu, Chris Douglas, heads up, this patch will set the default for ADLS access token provider type to ClientCredential.

        Show
        jzhuge John Zhuge added a comment - I am committing it. Atul Sikaria , Vishwajeet Dusane , Mingliang Liu , Chris Douglas , heads up, this patch will set the default for ADLS access token provider type to ClientCredential .
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for heads up. LGTM.

        Show
        liuml07 Mingliang Liu added a comment - Thanks for heads up. LGTM.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11548/)
        HADOOP-14174. Set default ADLS access token provider type to (jzhuge: rev 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5)

        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java
        • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11548/ ) HADOOP-14174 . Set default ADLS access token provider type to (jzhuge: rev 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5) (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        Hide
        jzhuge John Zhuge added a comment -

        Committed to trunk, branch-2, and branch-2.8.

        Thanks Lei (Eddy) Xu, Vishwajeet Dusane, Mingliang Liu for the review and comments.

        Show
        jzhuge John Zhuge added a comment - Committed to trunk, branch-2, and branch-2.8. Thanks Lei (Eddy) Xu , Vishwajeet Dusane , Mingliang Liu for the review and comments.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
        HADOOP-14174. Set default ADLS access token provider type to (jzhuge: rev 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5)

        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java
        • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java
        • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) HADOOP-14174 . Set default ADLS access token provider type to (jzhuge: rev 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            jzhuge John Zhuge
            Reporter:
            jzhuge John Zhuge
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development