Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: fs/adl
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      <!-- markdown -->

      * Properties {{dfs.adls.*}} are renamed {{fs.adl.*}}
      * Property {{adl.dfs.enable.client.latency.tracker}} is renamed {{adl.enable.client.latency.tracker}}
      * Old properties are still supported
      Show
      <!-- markdown --> * Properties {{dfs.adls.*}} are renamed {{fs.adl.*}} * Property {{adl.dfs.enable.client.latency.tracker}} is renamed {{adl.enable.client.latency.tracker}} * Old properties are still supported

      Description

      Rename properties with prefix dfs.adls. to fs.adl.
      Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker

      1. HADOOP-14038.001.patch
        12 kB
        John Zhuge
      2. HADOOP-14038.002.patch
        15 kB
        John Zhuge
      3. HADOOP-14038.003.patch
        16 kB
        John Zhuge
      4. HADOOP-14038.004.patch
        36 kB
        John Zhuge
      5. HADOOP-14038.005.patch
        16 kB
        John Zhuge
      6. HADOOP-14038.006.patch
        16 kB
        John Zhuge
      7. HADOOP-14038.007.patch
        15 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • Add credential properties to core-default.xml, make ClientCredential the default provider type
          • Fix unit tests that assume provider type is Custom
          • Update doc index.md

          Testing done

          • Live unit tests
          • Verify doc index.md
          Show
          jzhuge John Zhuge added a comment - Patch 001 Add credential properties to core-default.xml, make ClientCredential the default provider type Fix unit tests that assume provider type is Custom Update doc index.md Testing done Live unit tests Verify doc index.md
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002

          • Update section Protecting the Credentials with Credential Providers in index.md
          Show
          jzhuge John Zhuge added a comment - Patch 002 Update section Protecting the Credentials with Credential Providers in index.md
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 45s Maven dependency ordering for branch
          +1 mvninstall 14m 9s trunk passed
          +1 compile 13m 9s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 1m 57s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 12m 10s the patch passed
          +1 javac 12m 10s the patch passed
          +1 checkstyle 1m 55s the patch passed
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 17s the patch passed
          +1 javadoc 1m 16s the patch passed
          -1 unit 8m 16s hadoop-common in the patch failed.
          +1 unit 3m 41s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          93m 34s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853662/HADOOP-14038.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 59713b9822cf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8035749
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 14m 9s trunk passed +1 compile 13m 9s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 12m 10s the patch passed +1 javac 12m 10s the patch passed +1 checkstyle 1m 55s the patch passed +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 17s the patch passed +1 javadoc 1m 16s the patch passed -1 unit 8m 16s hadoop-common in the patch failed. +1 unit 3m 41s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 93m 34s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853662/HADOOP-14038.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 59713b9822cf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8035749 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11665/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 50s Maven dependency ordering for branch
          +1 mvninstall 11m 21s trunk passed
          +1 compile 11m 47s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 1m 15s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 15s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          +1 compile 10m 14s the patch passed
          +1 javac 10m 14s the patch passed
          +1 checkstyle 1m 46s the patch passed
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 1m 21s the patch passed
          -1 unit 7m 8s hadoop-common in the patch failed.
          +1 unit 3m 47s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          90m 7s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853669/HADOOP-14038.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3ed1705393e0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6ba61d2
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 50s Maven dependency ordering for branch +1 mvninstall 11m 21s trunk passed +1 compile 11m 47s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 1m 15s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 15s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 10m 14s the patch passed +1 javac 10m 14s the patch passed +1 checkstyle 1m 46s the patch passed +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 21s the patch passed -1 unit 7m 8s hadoop-common in the patch failed. +1 unit 3m 47s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 90m 7s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853669/HADOOP-14038.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3ed1705393e0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ba61d2 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11667/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 003

          • Fix TestCommonConfigurationFields
          Show
          jzhuge John Zhuge added a comment - Patch 003 Fix TestCommonConfigurationFields
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 13m 26s trunk passed
          +1 compile 14m 33s trunk passed
          +1 checkstyle 2m 14s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 2m 14s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 12m 25s the patch passed
          +1 javac 12m 25s the patch passed
          +1 checkstyle 2m 4s the patch passed
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 21s the patch passed
          +1 javadoc 1m 21s the patch passed
          -1 unit 8m 52s hadoop-common in the patch failed.
          +1 unit 5m 13s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          97m 51s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853762/HADOOP-14038.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 84647ad5e1e7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4804050
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 26s trunk passed +1 compile 14m 33s trunk passed +1 checkstyle 2m 14s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 14s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 12m 25s the patch passed +1 javac 12m 25s the patch passed +1 checkstyle 2m 4s the patch passed +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 21s the patch passed +1 javadoc 1m 21s the patch passed -1 unit 8m 52s hadoop-common in the patch failed. +1 unit 5m 13s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 97m 51s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853762/HADOOP-14038.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 84647ad5e1e7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4804050 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11675/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          TestKDiag passed locally for me.

          Show
          jzhuge John Zhuge added a comment - TestKDiag passed locally for me.
          Hide
          jzhuge John Zhuge added a comment -

          Vishwajeet Dusane and Atul Sikaria, could you please take a look at the patch? Please note the default access token provider type is switched from Custom to ClientCredential .

          Show
          jzhuge John Zhuge added a comment - Vishwajeet Dusane and Atul Sikaria , could you please take a look at the patch? Please note the default access token provider type is switched from Custom to ClientCredential .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Before this goes in, can someone explain why the prefix for this is "dfs.adl"? Every other filesystem uses the prefix "fs.$SCHEMA", which would make all of these "fs.adl." entries.

          Show
          stevel@apache.org Steve Loughran added a comment - Before this goes in, can someone explain why the prefix for this is "dfs.adl"? Every other filesystem uses the prefix "fs.$SCHEMA", which would make all of these "fs.adl." entries.
          Hide
          jzhuge John Zhuge added a comment -

          Probably a carryover of the old WebHDFS mindset. Since HADOOP-13037 rewrote the connector as an independent FileSystem, we should rename all dfs.adls. prefix to fs.adl.

          I am ok to do the substitutions in this JIRA. And rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker.

          Vishwajeet Dusane and Atul Sikaria, what do you think?

          Show
          jzhuge John Zhuge added a comment - Probably a carryover of the old WebHDFS mindset. Since HADOOP-13037 rewrote the connector as an independent FileSystem, we should rename all dfs.adls. prefix to fs.adl. I am ok to do the substitutions in this JIRA. And rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker . Vishwajeet Dusane and Atul Sikaria , what do you think?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Steve Loughran and John Zhuge - Thanks for pointing the gap with other file system w.r.t. schema. I agree we need to change dfs.adls prefix to fs.adls. Please do add dfs.adls in deprecated list to not break existing usage of the key.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Steve Loughran and John Zhuge - Thanks for pointing the gap with other file system w.r.t. schema. I agree we need to change dfs.adls prefix to fs.adls . Please do add dfs.adls in deprecated list to not break existing usage of the key.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I agree with tagging everything as deprecated, we don't want to break anyone's setup

          Show
          stevel@apache.org Steve Loughran added a comment - I agree with tagging everything as deprecated, we don't want to break anyone's setup
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          John Zhuge - Also take a look at HADOOP-14113. Patch has updated ADL Doc, You might need to merge your changes with HADOOP-14113.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - John Zhuge - Also take a look at HADOOP-14113 . Patch has updated ADL Doc, You might need to merge your changes with HADOOP-14113 .
          Hide
          jzhuge John Zhuge added a comment -

          Patch 004

          • Rename properties with prefix dfs.adls. to fs.adl.. Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker. Rename dfs.adl.test.contract.enable to fs.adl.test.contract.enable. Deprecate the old properties.
          • Switch fs.adl.oauth2.access.token.provider.type default from Custom to ClientCredential
          • Update doc index.md extensively
          • Add properties to core-site.xml
          • Remove test class TestValidateConfiguration because I find it useless
          • Remove unused TOKEN_PROVIDER_TYPE_CLIENT_CRED and ADL_EVENTS_TRACKING_SOURCE

          Testing done

          • Live unit tests
          • hadoop fs -ls adl://… in single node setup
          • Verify doc

          Open question

          • Don’t like the way AdlStorageConfiguration static initializer has to call AdlConfKeys.addDeprecatedKeys
          Show
          jzhuge John Zhuge added a comment - Patch 004 Rename properties with prefix dfs.adls. to fs.adl. . Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker . Rename dfs.adl.test.contract.enable to fs.adl.test.contract.enable . Deprecate the old properties. Switch fs.adl.oauth2.access.token.provider.type default from Custom to ClientCredential Update doc index.md extensively Add properties to core-site.xml Remove test class TestValidateConfiguration because I find it useless Remove unused TOKEN_PROVIDER_TYPE_CLIENT_CRED and ADL_EVENTS_TRACKING_SOURCE Testing done Live unit tests hadoop fs -ls adl://… in single node setup Verify doc Open question Don’t like the way AdlStorageConfiguration static initializer has to call AdlConfKeys.addDeprecatedKeys
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 11s trunk passed
          +1 compile 12m 37s trunk passed
          +1 checkstyle 1m 51s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 9s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 10m 30s the patch passed
          +1 javac 10m 30s the patch passed
          +1 checkstyle 1m 55s the patch passed
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 16s the patch passed
          -1 unit 8m 20s hadoop-common in the patch failed.
          +1 unit 3m 41s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          89m 52s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854834/HADOOP-14038.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 037963d33542 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fbfe86d
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 11s trunk passed +1 compile 12m 37s trunk passed +1 checkstyle 1m 51s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 9s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 10m 30s the patch passed +1 javac 10m 30s the patch passed +1 checkstyle 1m 55s the patch passed +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 16s the patch passed -1 unit 8m 20s hadoop-common in the patch failed. +1 unit 3m 41s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 89m 52s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854834/HADOOP-14038.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 037963d33542 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fbfe86d Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11720/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 004 turned into a big refactoring that I hate. Let me break it up.

          Show
          jzhuge John Zhuge added a comment - Patch 004 turned into a big refactoring that I hate. Let me break it up.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          John Zhuge & Steve Loughran Should we consider adding deprecated-keyinfo-site.xml configuration file to list all the deprecated keys either mapped to new keys or none in this file parallel to core-site.xml? This is to avoid modules explicitly need to invoke Configuration.addDeprecations?

          John Zhuge I am fine with the patch size, if Steve Loughran is also fine with the patch then you can breaking up the patch. Your call.

          Will confirm backward compatibility is ensured with Patch04 and revert back.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - John Zhuge & Steve Loughran Should we consider adding deprecated-keyinfo-site.xml configuration file to list all the deprecated keys either mapped to new keys or none in this file parallel to core-site.xml ? This is to avoid modules explicitly need to invoke Configuration.addDeprecations ? John Zhuge I am fine with the patch size, if Steve Loughran is also fine with the patch then you can breaking up the patch. Your call. Will confirm backward compatibility is ensured with Patch04 and revert back.
          Hide
          jzhuge John Zhuge added a comment -

          Vishwajeet Dusane Patch 004 has the following pieces:

          • Rename dfs.adl properties, can support old properties
          • Change default token provider type to ClientCredential, fixed a few broken unit tests
          • Rename dfs.adl.<hostname>.hostname and dfs.adl.<hostname>.mountpoint, can not support exist properties due to the <hostname part
          • Remove 2 unused properties
          • Obsolete TestValidateConfiguration
          • Extra update to index.md

          BTW, are you ok to see LATENCY_TRACKER_KEY set to adl.enable.client.latency.tracker? Or you'd like adl.feature.enable.client.latency.tracker or adl.fs.enable.client.latency.tracker? Suggestion?

          Are you ok to obsolete TestValidateConfiguration class?

          Show
          jzhuge John Zhuge added a comment - Vishwajeet Dusane Patch 004 has the following pieces: Rename dfs.adl properties, can support old properties Change default token provider type to ClientCredential, fixed a few broken unit tests Rename dfs.adl.<hostname>.hostname and dfs.adl.<hostname>.mountpoint , can not support exist properties due to the <hostname part Remove 2 unused properties Obsolete TestValidateConfiguration Extra update to index.md BTW, are you ok to see LATENCY_TRACKER_KEY set to adl.enable.client.latency.tracker ? Or you'd like adl.feature.enable.client.latency.tracker or adl.fs.enable.client.latency.tracker ? Suggestion? Are you ok to obsolete TestValidateConfiguration class?
          Hide
          jzhuge John Zhuge added a comment -

          Patch 005

          • Rename properties with prefix dfs.adls. to fs.adl.
          • Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker
          • Rename dfs.adl.test.contract.enable to fs.adl.test.contract.enable
          • Update doc index.md
          • Remove the useless test class TestValidateConfiguration

          Testing done

          • Passed live unit tests with mixed old and new properties in auth-keys.xml
          • Verified doc

          Follow-up JIRA

          • Switch fs.adl.oauth2.access.token.provider.type default from Custom to ClientCredential
          • Add properties with default values to core-default.xml
          • Remove unused TOKEN_PROVIDER_TYPE_CLIENT_CRED and ADL_EVENTS_TRACKING_SOURCE

          Vishwajeet Dusane, Lei (Eddy) Xu, please take a look this patch. Should be much cleaner than 004.

          Show
          jzhuge John Zhuge added a comment - Patch 005 Rename properties with prefix dfs.adls. to fs.adl. Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker Rename dfs.adl.test.contract.enable to fs.adl.test.contract.enable Update doc index.md Remove the useless test class TestValidateConfiguration Testing done Passed live unit tests with mixed old and new properties in auth-keys.xml Verified doc Follow-up JIRA Switch fs.adl.oauth2.access.token.provider.type default from Custom to ClientCredential Add properties with default values to core-default.xml Remove unused TOKEN_PROVIDER_TYPE_CLIENT_CRED and ADL_EVENTS_TRACKING_SOURCE Vishwajeet Dusane , Lei (Eddy) Xu , please take a look this patch. Should be much cleaner than 004.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 13m 16s trunk passed
          +1 compile 0m 15s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 17s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 22s trunk passed
          +1 javadoc 0m 11s trunk passed
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 12s the patch passed
          +1 javac 0m 12s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 14s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 9s the patch passed
          +1 unit 3m 29s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          21m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857264/HADOOP-14038.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 3140d1d0d97a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a96afae
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11799/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11799/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 13m 16s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 22s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 12s the patch passed +1 javac 0m 12s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 3m 29s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 21m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857264/HADOOP-14038.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 3140d1d0d97a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a96afae Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11799/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11799/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks John Zhuge, Quick glance over the patch5

          1.

                String hostNameProperty = "fs.adl." + hostname + ".hostname";
                String mountPointProperty = "fs.adl." + hostname + ".mountpoint";
          

          Existing client using dfs.adls for mount point would fail to initialize. one way to handle would be to add deprecatedHostNameProperty && deprecatedMountPointProperty constructed as well with dfs.adls prefix.

          2. Also we should add test to ensure backward compatibility and test to ensure existing keys name and default value is validated. TestValidateConfiguration was responsible for that.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks John Zhuge , Quick glance over the patch5 1. String hostNameProperty = "fs.adl." + hostname + ".hostname" ; String mountPointProperty = "fs.adl." + hostname + ".mountpoint" ; Existing client using dfs.adls for mount point would fail to initialize. one way to handle would be to add deprecatedHostNameProperty && deprecatedMountPointProperty constructed as well with dfs.adls prefix. 2. Also we should add test to ensure backward compatibility and test to ensure existing keys name and default value is validated. TestValidateConfiguration was responsible for that.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Vishwajeet Dusane for the quick review.

          1. Unfortunately I don't see any way to deprecate configuration properties who names contain dynamic portion. Are hostNameProperty and mountPointProperty being used by many customers? If yes, should we just keep the old names?

          2. Maybe I miss something? TestValidateConfiguration does not seem to test deprecation at all. It only forces whoever changes AdlConfKeys to change this test class. For example:

          AdlConfKeys
            public static final String AZURE_AD_CLIENT_ID_KEY = "dfs.adls.oauth2.client.id";
          
          TestValidateConfiguration
              Assert.assertEquals("dfs.adls.oauth2.client.id", AZURE_AD_CLIENT_ID_KEY);
          
          Show
          jzhuge John Zhuge added a comment - Thanks Vishwajeet Dusane for the quick review. 1. Unfortunately I don't see any way to deprecate configuration properties who names contain dynamic portion. Are hostNameProperty and mountPointProperty being used by many customers? If yes, should we just keep the old names? 2. Maybe I miss something? TestValidateConfiguration does not seem to test deprecation at all. It only forces whoever changes AdlConfKeys to change this test class. For example: AdlConfKeys public static final String AZURE_AD_CLIENT_ID_KEY = "dfs.adls.oauth2.client.id" ; TestValidateConfiguration Assert.assertEquals( "dfs.adls.oauth2.client.id" , AZURE_AD_CLIENT_ID_KEY);
          Hide
          jzhuge John Zhuge added a comment -

          Allen Wittenauer, Steve Loughran, and Chris Nauroth, Is there any way to deprecate properties with dynamic names, e.g. dfs.adls.<hostname>.hostname?

          Other than checking both old and new properties in code:

            value = conf.get(newProp);
            if (value == null) {
              value = conf.get(oldProp);
            }
          
          Show
          jzhuge John Zhuge added a comment - Allen Wittenauer , Steve Loughran , and Chris Nauroth , Is there any way to deprecate properties with dynamic names, e.g. dfs.adls.<hostname>.hostname ? Other than checking both old and new properties in code: value = conf.get(newProp); if (value == null ) { value = conf.get(oldProp); }
          Hide
          jzhuge John Zhuge added a comment -

          Discovered an issue related to the deprecation of ADLS properties (stored in core-site.xml) in Patch 005:

          • Configuration.addDeprecations is called in AdlFileSystem static initializer
          • The conf passed into AdlFileSystem#initialize is constructed in FsShell before AdlFileSystem static initializer is called

          Thus, a simple command hadoop fs -ls adl:// fails with Patch 005.

          Allen Wittenauer, Steve Loughran, and Chris Nauroth, is it ok to call conf.reloadConfiguration() in AdlFileSystem#initialize? It fixes the issue but means reloadConfiguration is called for every new instance. Is there a better solution?

          Show
          jzhuge John Zhuge added a comment - Discovered an issue related to the deprecation of ADLS properties (stored in core-site.xml) in Patch 005: Configuration.addDeprecations is called in AdlFileSystem static initializer The conf passed into AdlFileSystem#initialize is constructed in FsShell before AdlFileSystem static initializer is called Thus, a simple command hadoop fs -ls adl:// fails with Patch 005. Allen Wittenauer , Steve Loughran , and Chris Nauroth , is it ok to call conf.reloadConfiguration() in AdlFileSystem#initialize ? It fixes the issue but means reloadConfiguration is called for every new instance. Is there a better solution?
          Hide
          cnauroth Chris Nauroth added a comment -

          Is there any way to deprecate properties with dynamic names, e.g. dfs.adls.<hostname>.hostname?

          Sorry, no, I am not aware of any convenient way to achieve this (barring the possibility of putting new features into Configuration to support it).

          It fixes the issue but means reloadConfiguration is called for every new instance.

          I would prefer to avoid that because of the extra I/O reading the files and the extra XML parsing. I thought every Configuration instance was sharing the same static/global DeprecationContext, so therefore wherever in the code we add new deprecations, it would propagate down to all live instances. Am I missing something?

          Show
          cnauroth Chris Nauroth added a comment - Is there any way to deprecate properties with dynamic names, e.g. dfs.adls.<hostname>.hostname ? Sorry, no, I am not aware of any convenient way to achieve this (barring the possibility of putting new features into Configuration to support it). It fixes the issue but means reloadConfiguration is called for every new instance. I would prefer to avoid that because of the extra I/O reading the files and the extra XML parsing. I thought every Configuration instance was sharing the same static/global DeprecationContext , so therefore wherever in the code we add new deprecations, it would propagate down to all live instances. Am I missing something?
          Hide
          jzhuge John Zhuge added a comment - - edited

          Unlike Configuration.addDefaultResource, Configuration.addDeprecations does not propagate to all live instances.

          Should I enhance Configuration.addDeprecations to reload all existing instances, or just let ADL addDeprecatedKeys to do it?

          Show
          jzhuge John Zhuge added a comment - - edited Unlike Configuration.addDefaultResource , Configuration.addDeprecations does not propagate to all live instances. Should I enhance Configuration.addDeprecations to reload all existing instances, or just let ADL addDeprecatedKeys to do it?
          Hide
          jzhuge John Zhuge added a comment -

          Patch 006

          • Rename properties with prefix dfs.adls. to fs.adl.
          • Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker
          • Add Configuration.reloadExistingConfigurations
          • Add AdlConfKeys.addDeprecatedKeys that calls Configuration.addDeprecations then Configuration.reloadExistingConfigurations
          • Leave dfs.adl.test.contract.enable alone to minimize changes
          • Update doc index.md
          • Retire test class TestValidateConfiguration

          Testing done

          • Live unit tests with mixed old and new properties in auth-keys.xml
          • Verify doc
          Show
          jzhuge John Zhuge added a comment - Patch 006 Rename properties with prefix dfs.adls. to fs.adl. Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker Add Configuration.reloadExistingConfigurations Add AdlConfKeys.addDeprecatedKeys that calls Configuration.addDeprecations then Configuration.reloadExistingConfigurations Leave dfs.adl.test.contract.enable alone to minimize changes Update doc index.md Retire test class TestValidateConfiguration Testing done Live unit tests with mixed old and new properties in auth-keys.xml Verify doc
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 41s Maven dependency ordering for branch
          +1 mvninstall 12m 20s trunk passed
          -1 compile 7m 6s root in trunk failed.
          +1 checkstyle 2m 1s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 21s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          -1 compile 6m 31s root in the patch failed.
          -1 javac 6m 31s root in the patch failed.
          +1 checkstyle 2m 3s the patch passed
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 27s the patch passed
          +1 javadoc 1m 27s the patch passed
          -1 unit 8m 24s hadoop-common in the patch failed.
          +1 unit 3m 46s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 48s The patch does not generate ASF License warnings.
          82m 21s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 12c8f66c4b31 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4db9cc7
          Default Java 1.8.0_121
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-compile-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 12m 20s trunk passed -1 compile 7m 6s root in trunk failed. +1 checkstyle 2m 1s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 48s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 21s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed -1 compile 6m 31s root in the patch failed. -1 javac 6m 31s root in the patch failed. +1 checkstyle 2m 3s the patch passed +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 1m 27s the patch passed -1 unit 8m 24s hadoop-common in the patch failed. +1 unit 3m 46s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 48s The patch does not generate ASF License warnings. 82m 21s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 12c8f66c4b31 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4db9cc7 Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-compile-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11809/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          LGTLM; some minor tweaks needed before it's ready to go in

          Configuration#756
          if the log.debug() message is doing string construction, the condition should include a check for isDebug. Actually, as It may be useful to know that 0 configs were reloaded, consider just having the guard look at log level alone:

          if (LOG.isDebugEnabled()) {
           LOG.debug("Reloading " + cnt + " existing configurations");
          }
          

          TestAzureADTokenProvider should really be using GenericTestUtils to check the message, or at least rethrow/wrap the inner stack trace.

           if (!e.getMessage().contains(AZURE_AD_TOKEN_PROVIDER_CLASS_KEY))) throw e;
          

          Stack traces are too valuable to lose, especially on a jenkins run.

          Show
          stevel@apache.org Steve Loughran added a comment - LGTLM; some minor tweaks needed before it's ready to go in Configuration#756 if the log.debug() message is doing string construction, the condition should include a check for isDebug. Actually, as It may be useful to know that 0 configs were reloaded, consider just having the guard look at log level alone: if (LOG.isDebugEnabled()) { LOG.debug( "Reloading " + cnt + " existing configurations" ); } TestAzureADTokenProvider should really be using GenericTestUtils to check the message, or at least rethrow/wrap the inner stack trace. if (!e.getMessage().contains(AZURE_AD_TOKEN_PROVIDER_CLASS_KEY))) throw e; Stack traces are too valuable to lose, especially on a jenkins run.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          2. Maybe I miss something? TestValidateConfiguration does not seem to test deprecation at all. It only forces...

          Instead of retiring TestValidateConfiguration, we retain TestValidateConfiguration to ensure keys defined in AdlConfKeys are not mistakenly modified. Also i am proposing extend test in TestValidateConfiguration for backward compatibility check.

          Example :

          @Test		
           public void validateConfigurationKeys() {
          ...
          }
          
          @Test		
           public void validateBackwardCompitibility() {
                 // Verify for static & dynamic keys as well.
                 Configuration conf = new Configuration();
                 conf.add("dfs.adls.oauth2.client.id","XYZ");
                 Assert.assertEquals("XYZ", conf.get(AZURE_AD_CLIENT_ID_KEY));
          }
          
          

          I am aware of few client who are consuming dynamic properties at the moment so being extra cautious on not breaking them.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - 2. Maybe I miss something? TestValidateConfiguration does not seem to test deprecation at all. It only forces... Instead of retiring TestValidateConfiguration , we retain TestValidateConfiguration to ensure keys defined in AdlConfKeys are not mistakenly modified. Also i am proposing extend test in TestValidateConfiguration for backward compatibility check. Example : @Test public void validateConfigurationKeys() { ... } @Test public void validateBackwardCompitibility() { // Verify for static & dynamic keys as well. Configuration conf = new Configuration(); conf.add( "dfs.adls.oauth2.client.id" , "XYZ" ); Assert.assertEquals( "XYZ" , conf.get(AZURE_AD_CLIENT_ID_KEY)); } I am aware of few client who are consuming dynamic properties at the moment so being extra cautious on not breaking them.
          Hide
          jzhuge John Zhuge added a comment -

          Steve Loughran Will include your comment in the next patch.

          Vishwajeet Dusane Thanks for the review. Totally agree with you on that we should be careful not to break any existing code.

          It is a good idea to add what you suggested. It will cover one use case: Configuration#set is called with an old key, then conf is read with the new key.

          There is another use case when a config file with old keys is loaded. These are 2 different code paths in Configuration class. I will add an unit test for it as well.

          For TestValidateConfiguration, I will keep it then since HADOOP-13037 reviewed this class already even though IMHO the accidental modification is already mitigated by 3 measures:

          1. The properties are in a class called AdlConfKeys which indicates these are conf keys.
          2. The properties all have _KEY suffix to indicate these are conf keys.
          3. The property values are in the format of aa.bb.cc.dd, somewhat a hint of property names.
          Show
          jzhuge John Zhuge added a comment - Steve Loughran Will include your comment in the next patch. Vishwajeet Dusane Thanks for the review. Totally agree with you on that we should be careful not to break any existing code. It is a good idea to add what you suggested. It will cover one use case: Configuration#set is called with an old key, then conf is read with the new key. There is another use case when a config file with old keys is loaded. These are 2 different code paths in Configuration class. I will add an unit test for it as well. For TestValidateConfiguration, I will keep it then since HADOOP-13037 reviewed this class already even though IMHO the accidental modification is already mitigated by 3 measures: The properties are in a class called AdlConfKeys which indicates these are conf keys. The properties all have _KEY suffix to indicate these are conf keys. The property values are in the format of aa.bb.cc.dd , somewhat a hint of property names.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 007 (incoporated all comments by Steve and Vishwajeet)

          • Rename properties with prefix dfs.adls. to fs.adl.
          • Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker
          • Add Configuration.reloadExistingConfigurations
          • Add AdlConfKeys.addDeprecatedKeys that calls Configuration.addDeprecations and Configuration.reloadExistingConfigurations
          • Update doc index.md
          • Add test case testSetDeprecatedKeys and testLoadDeprecatedKeys to TestValidateConfiguration

          Testing done

          • Live unit tests with mixed old and new properties in auth-keys.xml
          • Verify doc
          Show
          jzhuge John Zhuge added a comment - Patch 007 (incoporated all comments by Steve and Vishwajeet) Rename properties with prefix dfs.adls. to fs.adl. Rename adl.dfs.enable.client.latency.tracker to adl.enable.client.latency.tracker Add Configuration.reloadExistingConfigurations Add AdlConfKeys.addDeprecatedKeys that calls Configuration.addDeprecations and Configuration.reloadExistingConfigurations Update doc index.md Add test case testSetDeprecatedKeys and testLoadDeprecatedKeys to TestValidateConfiguration Testing done Live unit tests with mixed old and new properties in auth-keys.xml Verify doc
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 53s Maven dependency ordering for branch
          +1 mvninstall 12m 57s trunk passed
          +1 compile 15m 28s trunk passed
          +1 checkstyle 1m 58s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 2m 4s trunk passed
          +1 javadoc 1m 12s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 14m 47s the patch passed
          +1 javac 14m 47s the patch passed
          +1 checkstyle 2m 0s the patch passed
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 1m 18s the patch passed
          -1 unit 7m 46s hadoop-common in the patch failed.
          +1 unit 3m 42s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          98m 50s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860376/HADOOP-14038.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 369fd3421eb6 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ab759e9
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 53s Maven dependency ordering for branch +1 mvninstall 12m 57s trunk passed +1 compile 15m 28s trunk passed +1 checkstyle 1m 58s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 2m 4s trunk passed +1 javadoc 1m 12s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 14m 47s the patch passed +1 javac 14m 47s the patch passed +1 checkstyle 2m 0s the patch passed +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 20s the patch passed +1 javadoc 1m 18s the patch passed -1 unit 7m 46s hadoop-common in the patch failed. +1 unit 3m 42s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 98m 50s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860376/HADOOP-14038.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 369fd3421eb6 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ab759e9 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11920/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Vishwajeet Dusane and Steve Loughran, could you please take a look at patch 007 which should have incorporated all comments?

          Show
          jzhuge John Zhuge added a comment - Vishwajeet Dusane and Steve Loughran , could you please take a look at patch 007 which should have incorporated all comments?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks for incorporating comment John Zhuge. I see in Path07 you have removed dynamic key(dfs.adls.<hostname>.mountpoint and dfs.adls.<hostname>.hostname) rename from Patch06, was that intentional?

          I ran Live test and did some basic test with patch07, LGTM.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks for incorporating comment John Zhuge . I see in Path07 you have removed dynamic key( dfs.adls.<hostname>.mountpoint and dfs.adls.<hostname>.hostname ) rename from Patch06, was that intentional? I ran Live test and did some basic test with patch07, LGTM.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Vishwajeet Dusane for such a quick reply!

          Yes, mountpoint and hostname changes were removed in order to simplify the patch. If desired, we can create a follow-up JIRA, more unit tests will be needed.

          Show
          jzhuge John Zhuge added a comment - Thanks Vishwajeet Dusane for such a quick reply! Yes, mountpoint and hostname changes were removed in order to simplify the patch. If desired, we can create a follow-up JIRA, more unit tests will be needed.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Sounds good to me to create separate JIRA for the same. +1 on Patch07.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Sounds good to me to create separate JIRA for the same. +1 on Patch07.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11479 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11479/)
          HADOOP-14038. Rename ADLS credential properties. Contributed by John (jzhuge: rev 253e3e78abb9ebe0e8103d25538ff07875b33c84)

          • (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11479 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11479/ ) HADOOP-14038 . Rename ADLS credential properties. Contributed by John (jzhuge: rev 253e3e78abb9ebe0e8103d25538ff07875b33c84) (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java
          Hide
          jzhuge John Zhuge added a comment - - edited

          Filed INFRA-13768 for the Hadoop-trunk-Commit failure.

          Show
          jzhuge John Zhuge added a comment - - edited Filed INFRA-13768 for the Hadoop-trunk-Commit failure.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development