Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13392

[Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: fs/adl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      OAuth2 configuration is optional in case of WebHdfsFileSystem configuration. However OAuth2 configuration should be default set to true when AdlFileSystem is configured.

      1. HADOOP-13392.001.patch
        2 kB
        Vishwajeet Dusane
      2. HADOOP-13392.002.patch
        6 kB
        Chris Douglas
      3. HADOOP-13392.003.patch
        5 kB
        Chris Douglas

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for patch 003. I agree with cloning the config instance instead of mutating the provided one. I have committed this to trunk and branch-3.0.0-alpha1. Vishwajeet Dusane and Chris Douglas, thank you for the patch.

        Show
        cnauroth Chris Nauroth added a comment - +1 for patch 003. I agree with cloning the config instance instead of mutating the provided one. I have committed this to trunk and branch-3.0.0-alpha1. Vishwajeet Dusane and Chris Douglas , thank you for the patch.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment - - edited

        +1 - Change worked for me.

        In case there is a config change after initialization, then the change would not reflect to Adl and webhdfs layer. Not sure if there is a use case like that?

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - - edited +1 - Change worked for me. In case there is a config change after initialization, then the change would not reflect to Adl and webhdfs layer. Not sure if there is a use case like that?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 25s trunk passed
        +1 compile 0m 18s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 22s trunk passed
        +1 mvneclipse 1m 40s trunk passed
        +1 findbugs 0m 35s trunk passed
        +1 javadoc 0m 15s trunk passed
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 19s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 31s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 2m 17s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        17m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819568/HADOOP-13392.003.patch
        JIRA Issue HADOOP-13392
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dce7bd1e02f2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dca6335
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10062/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10062/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 25s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 1m 40s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 31s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 2m 17s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 17m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819568/HADOOP-13392.003.patch JIRA Issue HADOOP-13392 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dce7bd1e02f2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dca6335 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10062/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10062/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Chris Douglas - I think that should work as well. I will try out and confirm.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Douglas - I think that should work as well. I will try out and confirm.
        Hide
        chris.douglas Chris Douglas added a comment -

        Missed the change to ADLConfKeys

        Show
        chris.douglas Chris Douglas added a comment - Missed the change to ADLConfKeys
        Hide
        chris.douglas Chris Douglas added a comment -

        Instead of overriding the value in the config (which might interfere with other clients created subsequently), this can clone the configuration if oauth is disabled. Would this alternative work?

        Show
        chris.douglas Chris Douglas added a comment - Instead of overriding the value in the config (which might interfere with other clients created subsequently), this can clone the configuration if oauth is disabled. Would this alternative work?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 14s trunk passed
        +1 compile 0m 18s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        +1 checkstyle 0m 9s the patch passed
        +1 mvnsite 0m 18s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 33s the patch passed
        +1 javadoc 0m 12s the patch passed
        +1 unit 2m 13s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        16m 3s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch
        JIRA Issue HADOOP-13392
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a47accd5e662 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 38128ba
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10035/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10035/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 14s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 2m 13s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 16m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch JIRA Issue HADOOP-13392 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a47accd5e662 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 38128ba Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10035/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10035/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 51s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 43s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 15s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        +1 checkstyle 0m 10s the patch passed
        +1 mvnsite 0m 17s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 30s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 2m 18s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        16m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch
        JIRA Issue HADOOP-13392
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 53beb65ebd63 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 38128ba
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10032/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10032/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 51s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 2m 18s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 16m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch JIRA Issue HADOOP-13392 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 53beb65ebd63 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 38128ba Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10032/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10032/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 40s trunk passed
        +1 compile 0m 15s trunk passed
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 19s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 14s the patch passed
        +1 compile 0m 13s the patch passed
        +1 javac 0m 13s the patch passed
        +1 checkstyle 0m 9s the patch passed
        +1 mvnsite 0m 16s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 2m 15s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        14m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch
        JIRA Issue HADOOP-13392
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1cf677ee53b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1c9d2ab
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10029/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10029/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 40s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 14s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 2m 15s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819103/HADOOP-13392.001.patch JIRA Issue HADOOP-13392 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1cf677ee53b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1c9d2ab Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10029/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10029/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        To add more context. Azure data lake can not be accessed without OAuth2 access token being in as Authorization header hence configuring dfs.webhdfs.oauth2.enabled to true is redundant and can be avoided during AdlFileSystem initialization.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - To add more context. Azure data lake can not be accessed without OAuth2 access token being in as Authorization header hence configuring dfs.webhdfs.oauth2.enabled to true is redundant and can be avoided during AdlFileSystem initialization.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Setup "dfs.webhdfs.oauth2.enabled" to true when PrivateAzureDataLakeFileSystem initialized.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Setup "dfs.webhdfs.oauth2.enabled" to true when PrivateAzureDataLakeFileSystem initialized.

          People

          • Assignee:
            vishwajeet.dusane Vishwajeet Dusane
            Reporter:
            vishwajeet.dusane Vishwajeet Dusane
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development