Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13956

Read ADLS credentials from Credential Provider

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 2.8.0, 3.0.0-alpha3
    • Component/s: fs/adl
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      The hadoop-azure-datalake file system now supports configuration of the Azure Data Lake Store account credentials using the standard Hadoop Credential Provider API. For details, please refer to the documentation on hadoop-azure-datalake and the Credential Provider API.
      Show
      The hadoop-azure-datalake file system now supports configuration of the Azure Data Lake Store account credentials using the standard Hadoop Credential Provider API. For details, please refer to the documentation on hadoop-azure-datalake and the Credential Provider API.

      Description

      Read ADLS credentials using Hadoop CredentialProvider API. See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html.

      1. HADOOP-13956.001.patch
        8 kB
        John Zhuge
      2. HADOOP-13956.002.patch
        12 kB
        John Zhuge
      3. HADOOP-13956.003.patch
        12 kB
        John Zhuge
      4. HADOOP-13956.004.patch
        14 kB
        John Zhuge
      5. HADOOP-13956.005.patch
        16 kB
        John Zhuge
      6. HADOOP-13956.006.patch
        16 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Good proposal, John Zhuge!
          One thing to keep in mind is something similar to HADOOP-12846, HADOOP-13548 may pop up.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Good proposal, John Zhuge ! One thing to keep in mind is something similar to HADOOP-12846 , HADOOP-13548 may pop up.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • Call getPassword for property dfs.adls.oauth2.credential and dfs.adls.oauth2.refresh.token
          • Add unit tests
          Show
          jzhuge John Zhuge added a comment - Patch 001 Call getPassword for property dfs.adls.oauth2.credential and dfs.adls.oauth2.refresh.token Add unit tests
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 53s trunk passed
          +1 compile 0m 15s trunk passed
          +1 checkstyle 0m 11s trunk passed
          +1 mvnsite 0m 16s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 23s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 13s the patch passed
          +1 javac 0m 13s the patch passed
          -0 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
          +1 mvnsite 0m 13s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 26s the patch passed
          +1 javadoc 0m 9s the patch passed
          +1 unit 3m 33s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          21m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847736/HADOOP-13956.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 835fa75e1d8f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e407449
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 53s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 16s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed -0 checkstyle 0m 9s hadoop-tools/hadoop-azure-datalake: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) +1 mvnsite 0m 13s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 26s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 3m 33s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 21m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847736/HADOOP-13956.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 835fa75e1d8f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e407449 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11445/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002

          • Add docs to CredentialProviderAPI.md and index.md
          • Fix checkstyle
          Show
          jzhuge John Zhuge added a comment - Patch 002 Add docs to CredentialProviderAPI.md and index.md Fix checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 12m 34s trunk passed
          +1 compile 9m 38s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 8s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 9m 13s the patch passed
          +1 javac 9m 13s the patch passed
          -0 checkstyle 1m 34s root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 15s the patch passed
          +1 unit 8m 59s hadoop-common in the patch passed.
          +1 unit 3m 48s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          84m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847866/HADOOP-13956.002.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 74ff263cfa64 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 78b487b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 12m 34s trunk passed +1 compile 9m 38s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 9m 13s the patch passed +1 javac 9m 13s the patch passed -0 checkstyle 1m 34s root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 8m 59s hadoop-common in the patch passed. +1 unit 3m 48s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 84m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847866/HADOOP-13956.002.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 74ff263cfa64 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 78b487b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11448/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 003

          • Fix checkstyle
          Show
          jzhuge John Zhuge added a comment - Patch 003 Fix checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 14m 32s trunk passed
          +1 compile 11m 18s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 1s the patch passed
          +1 compile 11m 4s the patch passed
          +1 javac 11m 4s the patch passed
          +1 checkstyle 1m 48s the patch passed
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 53s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 8m 36s hadoop-common in the patch passed.
          +1 unit 3m 50s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          91m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847886/HADOOP-13956.003.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 0e4de99f0dae 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 78b487b
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11449/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11449/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 14m 32s trunk passed +1 compile 11m 18s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 11m 4s the patch passed +1 javac 11m 4s the patch passed +1 checkstyle 1m 48s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 53s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 8m 36s hadoop-common in the patch passed. +1 unit 3m 50s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 91m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847886/HADOOP-13956.003.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 0e4de99f0dae 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 78b487b Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11449/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11449/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 004

          • Call getPassword for property dfs.adls.oauth2.client.id and dfs.adls.oauth2.refresh.url as well because index.md says client id should be kept secret and we might as well include refresh url which contains tenant id.
          • Add 2 more unit tests for credential provider fallback to configuration file
          Show
          jzhuge John Zhuge added a comment - Patch 004 Call getPassword for property dfs.adls.oauth2.client.id and dfs.adls.oauth2.refresh.url as well because index.md says client id should be kept secret and we might as well include refresh url which contains tenant id. Add 2 more unit tests for credential provider fallback to configuration file
          Hide
          jzhuge John Zhuge added a comment -

          TODO: add code similar to HADOOP-12846 as Wei-Chiu Chuang suggested

          Show
          jzhuge John Zhuge added a comment - TODO: add code similar to HADOOP-12846 as Wei-Chiu Chuang suggested
          Hide
          jzhuge John Zhuge added a comment -

          Patch 005

          • Call excludeIncompatibleCredentialProviders before getting any credential property
          • Add unit tests for excludeIncompatibleCredentialProviders
          Show
          jzhuge John Zhuge added a comment - Patch 005 Call excludeIncompatibleCredentialProviders before getting any credential property Add unit tests for excludeIncompatibleCredentialProviders
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 1s Maven dependency ordering for branch
          +1 mvninstall 13m 57s trunk passed
          +1 compile 11m 27s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 9s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 9m 27s the patch passed
          +1 javac 9m 27s the patch passed
          -0 checkstyle 1m 40s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 36s the patch passed
          +1 javadoc 1m 14s the patch passed
          +1 unit 8m 46s hadoop-common in the patch passed.
          +1 unit 3m 43s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          89m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848180/HADOOP-13956.004.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 9d7d087c296f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2bf4c6e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 1s Maven dependency ordering for branch +1 mvninstall 13m 57s trunk passed +1 compile 11m 27s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 9s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 9m 27s the patch passed +1 javac 9m 27s the patch passed -0 checkstyle 1m 40s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 36s the patch passed +1 javadoc 1m 14s the patch passed +1 unit 8m 46s hadoop-common in the patch passed. +1 unit 3m 43s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 89m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848180/HADOOP-13956.004.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 9d7d087c296f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2bf4c6e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11462/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 53s Maven dependency ordering for branch
          +1 mvninstall 12m 56s trunk passed
          +1 compile 10m 10s trunk passed
          +1 checkstyle 1m 34s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 16s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 4s the patch passed
          +1 compile 10m 33s the patch passed
          +1 javac 10m 33s the patch passed
          -0 checkstyle 1m 37s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 26s the patch passed
          +1 javadoc 1m 27s the patch passed
          +1 unit 10m 2s hadoop-common in the patch passed.
          +1 unit 4m 3s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 45s The patch does not generate ASF License warnings.
          90m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848183/HADOOP-13956.005.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 6567e0b9543d 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2bf4c6e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 53s Maven dependency ordering for branch +1 mvninstall 12m 56s trunk passed +1 compile 10m 10s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 10m 33s the patch passed +1 javac 10m 33s the patch passed -0 checkstyle 1m 37s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 26s the patch passed +1 javadoc 1m 27s the patch passed +1 unit 10m 2s hadoop-common in the patch passed. +1 unit 4m 3s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 45s The patch does not generate ASF License warnings. 90m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848183/HADOOP-13956.005.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 6567e0b9543d 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2bf4c6e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11463/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 006

          • Fix checkstyle
          Show
          jzhuge John Zhuge added a comment - Patch 006 Fix checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 9s Maven dependency ordering for branch
          +1 mvninstall 14m 59s trunk passed
          +1 compile 10m 37s trunk passed
          +1 checkstyle 1m 35s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 9m 59s the patch passed
          +1 javac 9m 59s the patch passed
          +1 checkstyle 1m 37s the patch passed
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 1m 16s the patch passed
          +1 unit 8m 1s hadoop-common in the patch passed.
          +1 unit 3m 45s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          90m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13956
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848206/HADOOP-13956.006.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 4498ea98907e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 383aa9c
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11464/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11464/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 9s Maven dependency ordering for branch +1 mvninstall 14m 59s trunk passed +1 compile 10m 37s trunk passed +1 checkstyle 1m 35s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 9m 59s the patch passed +1 javac 9m 59s the patch passed +1 checkstyle 1m 37s the patch passed +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 25s the patch passed +1 javadoc 1m 16s the patch passed +1 unit 8m 1s hadoop-common in the patch passed. +1 unit 3m 45s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 90m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13956 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848206/HADOOP-13956.006.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 4498ea98907e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 383aa9c Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11464/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11464/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hey, John Zhuge

          The patch looks good overall.

          I ran mvn test under hadoop-tools/hadoop-azure-datalake, the following tests failed:

          Failed tests:
            TestAdlContractRootDirLive>AbstractContractRootDirectoryTest.testRmNonEmptyRootDirNonRecursive:132->Assert.fail:88 non recursive delete should have raised an exception, but completed with exit code false
          

          Is it relevant to this patch?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hey, John Zhuge The patch looks good overall. I ran mvn test under hadoop-tools/hadoop-azure-datalake , the following tests failed: Failed tests: TestAdlContractRootDirLive>AbstractContractRootDirectoryTest.testRmNonEmptyRootDirNonRecursive:132->Assert.fail:88 non recursive delete should have raised an exception, but completed with exit code false Is it relevant to this patch?
          Hide
          jzhuge John Zhuge added a comment -

          The unit test failure is tracked by HADOOP-13927 "ADLS TestAdlContractRootDirLive.testRmNonEmptyRootDirNonRecursive failed". It is confirmed to be an ADLS backend bug.

          Show
          jzhuge John Zhuge added a comment - The unit test failure is tracked by HADOOP-13927 "ADLS TestAdlContractRootDirLive.testRmNonEmptyRootDirNonRecursive failed". It is confirmed to be an ADLS backend bug.
          Hide
          jzhuge John Zhuge added a comment -

          Deployed patch 006 to a 4-node cluster, and set up JCE keystore on HDFS. Everything works well.

          core-site.xml must has these configured:

          <property>
            <name>hadoop.security.credential.provider.path</name>
            <value>jceks://hdfs/cdep/keystores/creds.jceks</value>
          </property>
          <property>
            <name>dfs.adls.oauth2.access.token.provider.type</name>
            <value>ClientCredential</value>
          <property>
          

          Run these commands to populate the keystore:

          hadoop credential create dfs.adls.oauth2.client.id -value '123'
          hadoop credential create dfs.adls.oauth2.credential -value '456'
          hadoop credential create dfs.adls.oauth2.refresh.url -value '789'
          

          Unfortunately dfs.adls.oauth2.access.token.provider.type can not be easily put into the keystore because Configuration#getEnum is used to get this property. Anyway it is not really a secret.

          Show
          jzhuge John Zhuge added a comment - Deployed patch 006 to a 4-node cluster, and set up JCE keystore on HDFS. Everything works well. core-site.xml must has these configured: <property> <name>hadoop.security.credential.provider.path</name> <value>jceks: //hdfs/cdep/keystores/creds.jceks</value> </property> <property> <name>dfs.adls.oauth2.access.token.provider.type</name> <value>ClientCredential</value> <property> Run these commands to populate the keystore: hadoop credential create dfs.adls.oauth2.client.id -value '123' hadoop credential create dfs.adls.oauth2.credential -value '456' hadoop credential create dfs.adls.oauth2.refresh.url -value '789' Unfortunately dfs.adls.oauth2.access.token.provider.type can not be easily put into the keystore because Configuration#getEnum is used to get this property. Anyway it is not really a secret.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          That makes sense, John Zhuge. Thanks.

          +1. Committed to trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - That makes sense, John Zhuge . Thanks. +1. Committed to trunk.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Lei (Eddy) Xu for the review and commit! Thanks Wei-Chiu Chuang for the review!

          Show
          jzhuge John Zhuge added a comment - Thanks Lei (Eddy) Xu for the review and commit! Thanks Wei-Chiu Chuang for the review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11153 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11153/)
          HADOOP-13956. Read ADLS credentials from Credential Provider. (John (lei: rev e015b563197a475e354bf84fd27e7bbcc67e00a4)

          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md
          • (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11153 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11153/ ) HADOOP-13956 . Read ADLS credentials from Credential Provider. (John (lei: rev e015b563197a475e354bf84fd27e7bbcc67e00a4) (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development