Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13037

Refactor Azure Data Lake Store as an independent FileSystem

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/adl
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hadoop now supports integration with Azure Data Lake as an alternative Hadoop-compatible file system. Please refer to the Hadoop site documentation of Azure Data Lake for details on usage and configuration.

      Description

      The jira proposes an improvement over HADOOP-12666 to remove webhdfs dependencies from the ADL file system client and build out a standalone client. At a high level, this approach would extend the Hadoop file system class to provide an implementation for accessing Azure Data Lake. The scheme used for accessing the file system will continue to be

      adl://<accountname>.azuredatalake.net/path/to/file.

      The Azure Data Lake Cloud Store will continue to provide a webHDFS rest interface. The client will access the ADLS store using WebHDFS Rest APIs provided by the ADLS store.

      1. HADOOP-13037.005.patch
        314 kB
        Chris Douglas
      2. HADOOP-13037.006.patch
        314 kB
        Vishwajeet Dusane
      3. HADOOP-13037-001.patch
        390 kB
        Vishwajeet Dusane
      4. HADOOP-13037-002.patch
        396 kB
        Vishwajeet Dusane
      5. HADOOP-13037-003.patch
        397 kB
        Vishwajeet Dusane
      6. HADOOP-13037-004.patch
        317 kB
        Vishwajeet Dusane
      7. HADOOP-13037 Proposal.pdf
        319 kB
        Vishwajeet Dusane

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          I have created a new branch in git and a new fix version in JIRA, both named HADOOP-13037. We can continue development on that branch. Please track patches going into the feature branch using JIRA sub-tasks with the target version set to "HADOOP-13037".

          Show
          cnauroth Chris Nauroth added a comment - I have created a new branch in git and a new fix version in JIRA, both named HADOOP-13037 . We can continue development on that branch. Please track patches going into the feature branch using JIRA sub-tasks with the target version set to " HADOOP-13037 ".
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks Chris Nauroth, we will follow the process of creating sub-tasks and push change set in HADOOP-13037 branch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks Chris Nauroth , we will follow the process of creating sub-tasks and push change set in HADOOP-13037 branch.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          The existing ADLS FileSystem implementation and contract tests were committed in HADOOP-12666 and related subtasks. HADOOP-13037 proposes to replace this implementation with one that leverages capabilities of the underlying store and eliminates dependencies on WebHDFS.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - The existing ADLS FileSystem implementation and contract tests were committed in HADOOP-12666 and related subtasks. HADOOP-13037 proposes to replace this implementation with one that leverages capabilities of the underlying store and eliminates dependencies on WebHDFS.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Updating first draft of the change to get early start on the review.

          • The dependency Adl SDK in process to get submitted in to Maven repository. Once the submission is through, will update this JIRA.
          • We have added additional test cases on top of existing test submitted part of the HADOOP-12875
          • I have also incorporated review comments from the HADOOP-13257.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Updating first draft of the change to get early start on the review. The dependency Adl SDK in process to get submitted in to Maven repository. Once the submission is through, will update this JIRA. We have added additional test cases on top of existing test submitted part of the HADOOP-12875 I have also incorporated review comments from the HADOOP-13257 .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Trigger Hadoop QA build on the patch since the test-patch script is failing to build hadoop-common code. However compilation succeed without test-patch script. I will look into test-patch issue.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Trigger Hadoop QA build on the patch since the test-patch script is failing to build hadoop-common code. However compilation succeed without test-patch script. I will look into test-patch issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-13037 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833170/HADOOP-13037-001.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10768/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-13037 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833170/HADOOP-13037-001.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10768/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          It looks like v001 accidentally reverts HDFS-10789.

          Show
          chris.douglas Chris Douglas added a comment - It looks like v001 accidentally reverts HDFS-10789 .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks Chris Douglas, i think i generated patch against trunk from feature branch hence it pulled all the changeset from trunk. I will regenerate and submit patch against trunk only.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks Chris Douglas , i think i generated patch against trunk from feature branch hence it pulled all the changeset from trunk. I will regenerate and submit patch against trunk only.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -
          • The dependency Adl SDK in process to get submitted in to Maven repository. Once the submission is through, will update this JIRA.
          • We have added additional test cases on top of existing test submitted part of the HADOOP-12875
          • I have also incorporated review comments from the HADOOP-13257.

          As suggested by Chris Douglas - Updated patch against trunk instead of feature branch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - The dependency Adl SDK in process to get submitted in to Maven repository. Once the submission is through, will update this JIRA. We have added additional test cases on top of existing test submitted part of the HADOOP-12875 I have also incorporated review comments from the HADOOP-13257 . As suggested by Chris Douglas - Updated patch against trunk instead of feature branch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 51 new or modified test files.
          +1 mvninstall 8m 54s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 11s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 13s trunk passed
          -1 mvninstall 0m 7s hadoop-azure-datalake in the patch failed.
          -1 compile 0m 7s hadoop-azure-datalake in the patch failed.
          -1 javac 0m 7s hadoop-azure-datalake in the patch failed.
          +1 checkstyle 0m 6s the patch passed
          -1 mvnsite 0m 6s hadoop-azure-datalake in the patch failed.
          -1 mvneclipse 0m 6s hadoop-azure-datalake in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          -1 findbugs 0m 7s hadoop-azure-datalake in the patch failed.
          -1 javadoc 0m 6s hadoop-azure-datalake in the patch failed.
          -1 unit 0m 7s hadoop-azure-datalake in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          13m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833332/HADOOP-13037-001.patch
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux bb9edc29ca88 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbe663d
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-azure-datalake.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-azure-datalake.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-azure-datalake.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/testReport/
          modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 51 new or modified test files. +1 mvninstall 8m 54s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 13s trunk passed -1 mvninstall 0m 7s hadoop-azure-datalake in the patch failed. -1 compile 0m 7s hadoop-azure-datalake in the patch failed. -1 javac 0m 7s hadoop-azure-datalake in the patch failed. +1 checkstyle 0m 6s the patch passed -1 mvnsite 0m 6s hadoop-azure-datalake in the patch failed. -1 mvneclipse 0m 6s hadoop-azure-datalake in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. -1 findbugs 0m 7s hadoop-azure-datalake in the patch failed. -1 javadoc 0m 6s hadoop-azure-datalake in the patch failed. -1 unit 0m 7s hadoop-azure-datalake in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 13m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833332/HADOOP-13037-001.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux bb9edc29ca88 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbe663d Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-azure-datalake.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-compile-hadoop-tools_hadoop-azure-datalake.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-azure-datalake.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10786/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          Hi Vishwajeet Dusane. Thanks for doing this work. Is there source available for the new SDK this depends on? I was reading the patch and wanted to look at the create() logic, for example.

          Show
          fabbri Aaron Fabbri added a comment - Hi Vishwajeet Dusane . Thanks for doing this work. Is there source available for the new SDK this depends on? I was reading the patch and wanted to look at the create() logic, for example.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Resubmitting patch with dependent JAVA SDK snapshot path https://oss.sonatype.org/content/repositories/snapshots/com/microsoft/azure/azure-data-lake-store-sdk/
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Aaron Fabbri - You can browse through detail implementation under https://github.com/Azure/azure-data-lake-store-java repository.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Aaron Fabbri - You can browse through detail implementation under https://github.com/Azure/azure-data-lake-store-java repository.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Reverting patch since Jenkins Hadoop QA build did not trigger on the patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Reverting patch since Jenkins Hadoop QA build did not trigger on the patch.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Resubmit hadoop patch to trigger jenkins build.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Resubmit hadoop patch to trigger jenkins build.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 51 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 50s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 34s Maven dependency ordering for patch
          -1 mvninstall 0m 12s hadoop-azure-datalake in the patch failed.
          -1 compile 6m 47s root in the patch failed.
          -1 javac 6m 47s root in the patch failed.
          +1 checkstyle 1m 33s the patch passed
          -1 mvnsite 0m 16s hadoop-azure-datalake in the patch failed.
          +1 mvneclipse 0m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          -1 findbugs 0m 19s hadoop-azure-datalake in the patch failed.
          -1 javadoc 0m 17s hadoop-tools_hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 unit 10m 0s hadoop-common in the patch passed.
          -1 unit 0m 18s hadoop-azure-datalake in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          69m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835364/HADOOP-13037-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 40a63bc10527 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e90af4a
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-compile-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-datalake.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 51 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 6m 50s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 34s Maven dependency ordering for patch -1 mvninstall 0m 12s hadoop-azure-datalake in the patch failed. -1 compile 6m 47s root in the patch failed. -1 javac 6m 47s root in the patch failed. +1 checkstyle 1m 33s the patch passed -1 mvnsite 0m 16s hadoop-azure-datalake in the patch failed. +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. -1 findbugs 0m 19s hadoop-azure-datalake in the patch failed. -1 javadoc 0m 17s hadoop-tools_hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 unit 10m 0s hadoop-common in the patch passed. -1 unit 0m 18s hadoop-azure-datalake in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 69m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835364/HADOOP-13037-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 40a63bc10527 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e90af4a Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-compile-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-datalake.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10901/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Packaged AdlPermission.java within patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Packaged AdlPermission.java within patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 51 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 8m 37s trunk passed
          +1 compile 7m 24s trunk passed
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 1m 21s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          -1 compile 7m 17s root in the patch failed.
          -1 javac 7m 17s root in the patch failed.
          +1 checkstyle 1m 31s the patch passed
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          -1 findbugs 0m 19s hadoop-azure-datalake in the patch failed.
          -1 javadoc 0m 17s hadoop-tools_hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 unit 8m 26s hadoop-common in the patch passed.
          -1 unit 2m 34s hadoop-azure-datalake in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          72m 44s



          Reason Tests
          Failed junit tests hadoop.fs.adl.TestAdlRead
            hadoop.fs.adl.live.TestAdlSupportedCharsetInPath



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835383/HADOOP-13037-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3287813d38c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1f8490a
          Default Java 1.8.0_101
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-compile-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-datalake.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 51 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 8m 37s trunk passed +1 compile 7m 24s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed -1 compile 7m 17s root in the patch failed. -1 javac 7m 17s root in the patch failed. +1 checkstyle 1m 31s the patch passed +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. -1 findbugs 0m 19s hadoop-azure-datalake in the patch failed. -1 javadoc 0m 17s hadoop-tools_hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 unit 8m 26s hadoop-common in the patch passed. -1 unit 2m 34s hadoop-azure-datalake in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 72m 44s Reason Tests Failed junit tests hadoop.fs.adl.TestAdlRead   hadoop.fs.adl.live.TestAdlSupportedCharsetInPath Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835383/HADOOP-13037-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3287813d38c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f8490a Default Java 1.8.0_101 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-compile-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-datalake.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10902/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Patch compilation is failing to resolve SDK dependency with error

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hadoop-tools-dist: Failed to resolve dependencies for one or more projects in the reactor. Reason: No versions are present in the repository for the artifact with a range [2.0,)
          [ERROR] com.microsoft.azure:azure-data-lake-store-sdk:jar:null
          [ERROR] 
          [ERROR] from the specified remote repositories:
          [ERROR] apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots, releases=true, snapshots=true),
          [ERROR] repository.jboss.org (http://repository.jboss.org/nexus/content/groups/public/, releases=true, snapshots=false),
          [ERROR] central (http://repo.maven.apache.org/maven2, releases=true, snapshots=false),
          [ERROR] snapshots-repo (https://oss.sonatype.org/content/repositories/snapshots, releases=false, snapshots=true)
          [ERROR] Path to dependency:
          [ERROR] 1) org.apache.hadoop:hadoop-tools-dist:jar:3.0.0-alpha2-SNAPSHOT
          [ERROR] 2) org.apache.hadoop:hadoop-azure-datalake:jar:3.0.0-alpha2-SNAPSHOT
          [ERROR] -> [Help 1]
          [ERROR] 
          

          However with test-patch script, Patch does pass through compilation.

          Instead of passing range version, added a fixed version for SDK. Resubmitting the patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Patch compilation is failing to resolve SDK dependency with error [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process ( default ) on project hadoop-tools-dist: Failed to resolve dependencies for one or more projects in the reactor. Reason: No versions are present in the repository for the artifact with a range [2.0,) [ERROR] com.microsoft.azure:azure-data-lake-store-sdk:jar: null [ERROR] [ERROR] from the specified remote repositories: [ERROR] apache.snapshots.https (https: //repository.apache.org/content/repositories/snapshots, releases= true , snapshots= true ), [ERROR] repository.jboss.org (http: //repository.jboss.org/nexus/content/groups/ public /, releases= true , snapshots= false ), [ERROR] central (http: //repo.maven.apache.org/maven2, releases= true , snapshots= false ), [ERROR] snapshots-repo (https: //oss.sonatype.org/content/repositories/snapshots, releases= false , snapshots= true ) [ERROR] Path to dependency: [ERROR] 1) org.apache.hadoop:hadoop-tools-dist:jar:3.0.0-alpha2-SNAPSHOT [ERROR] 2) org.apache.hadoop:hadoop-azure-datalake:jar:3.0.0-alpha2-SNAPSHOT [ERROR] -> [Help 1] [ERROR] However with test-patch script, Patch does pass through compilation. Instead of passing range version, added a fixed version for SDK. Resubmitting the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 51 new or modified test files.
          0 mvndep 1m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 55s trunk passed
          +1 compile 7m 35s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 1m 7s the patch passed
          +1 compile 8m 23s the patch passed
          -1 javac 8m 23s root generated 1 new + 703 unchanged - 0 fixed = 704 total (was 703)
          -0 checkstyle 1m 35s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          -1 findbugs 0m 35s hadoop-tools/hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 4s the patch passed
          -1 unit 20m 25s hadoop-common in the patch failed.
          -1 unit 3m 57s hadoop-azure-datalake in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          87m 29s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-azure-datalake
            org.apache.hadoop.fs.adl.AdlPermission doesn't override org.apache.hadoop.fs.permission.FsPermission.equals(Object) At AdlPermission.java:At AdlPermission.java:[line 1]
          Failed junit tests hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835494/HADOOP-13037-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 68dfa0aebd7a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a895c66
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure-datalake.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 51 new or modified test files. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 7m 35s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 8m 23s the patch passed -1 javac 8m 23s root generated 1 new + 703 unchanged - 0 fixed = 704 total (was 703) -0 checkstyle 1m 35s root: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. -1 findbugs 0m 35s hadoop-tools/hadoop-azure-datalake generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 4s the patch passed -1 unit 20m 25s hadoop-common in the patch failed. -1 unit 3m 57s hadoop-azure-datalake in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 87m 29s Reason Tests FindBugs module:hadoop-tools/hadoop-azure-datalake   org.apache.hadoop.fs.adl.AdlPermission doesn't override org.apache.hadoop.fs.permission.FsPermission.equals(Object) At AdlPermission.java:At AdlPermission.java: [line 1] Failed junit tests hadoop.fs.adl.live.TestAdlSupportedCharsetInPath Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835494/HADOOP-13037-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 68dfa0aebd7a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a895c66 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure-datalake.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10906/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Fixed Findbugs, Checkstyle and JUnit issues.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Fixed Findbugs, Checkstyle and JUnit issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 51 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 6m 56s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 6m 50s the patch passed
          +1 javac 6m 50s the patch passed
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 1m 4s the patch passed
          -1 unit 20m 1s hadoop-common in the patch failed.
          +1 unit 3m 44s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          82m 34s



          Reason Tests
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835602/HADOOP-13037-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 7823266a2f09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ac35ee9
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 51 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 6m 56s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 6m 50s the patch passed +1 javac 6m 50s the patch passed +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 findbugs 2m 4s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 20m 1s hadoop-common in the patch failed. +1 unit 3m 44s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 82m 34s Reason Tests Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835602/HADOOP-13037-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 7823266a2f09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac35ee9 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10910/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          As a heads-up, my HADOOP-13687 patch is restructuring hadoop-azure-datalake under a new hadoop-cloud-storage module, which will serve as a unified dependency artifact that downstream projects can use to pick up all cloud storage file systems that ship within Hadoop. Before I commit HADOOP-13687, I'm going to prepare a rebased version of the HADOOP-13037 patch. This will just be a mechanical change of the source file paths in the patch, probably mostly automated with sed.

          Show
          cnauroth Chris Nauroth added a comment - As a heads-up, my HADOOP-13687 patch is restructuring hadoop-azure-datalake under a new hadoop-cloud-storage module, which will serve as a unified dependency artifact that downstream projects can use to pick up all cloud storage file systems that ship within Hadoop. Before I commit HADOOP-13687 , I'm going to prepare a rebased version of the HADOOP-13037 patch. This will just be a mechanical change of the source file paths in the patch, probably mostly automated with sed.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks for working through the build issues, and following up on the review feedback Vishwajeet Dusane.

          This may not require a branch. While the patch is large, much of it is removing code (to sever the WebHDFS dependency) and adding new tests. The former can be committed directly to trunk, and the latter we can commit in one or more followup issues (as subtasks of HADOOP-13257?). Any objections to this approach?

          A few minor comments from skimming the latest patch:

          • AccessTokenProviderCache uses an unusual singleton pattern. Does it need to create a private instance of itself, rather than initializing a static map directly?
          • AdlPermission#equals and #hashcode implementations are identical to what they override. Instead of reading from the config to report getAclBit (which is expensive, and unstable) it could either store the boolean or return basic FsPermission instances when disabled (and conversely, AdlPermission can always return true when the config knob is set). That this is a configurable boolean is surprising, and could be explained in this class's (empty) javadoc.
          • TokenProviderType::fromString is unused
          • In initialize, accountFQDN init can move up, so the accessTokenProvider init can be contiguous
          • Both the subclasses of AzureADTokenProvider throw exceptions for all abstract methods save init, and appear not to use the private instance they configure. The init pattern here is pretty complicated, with initialize creating an instance and adding it to a global cache, and not using the reference it retains. Can this be simplified? If the instance were returned, AccessTokenProviderCache additions can be moved to AdlFileSystem, so all the get/add calls are in one class.
          • The TokenProviderCache type seems unnecessary, if AzureADTP instances can return null to prevent caching (instead of the instanceof/cast)
          • It's a matter of taste, but importing all the static members of AdlConfKeys in AdlFileSystem (or none of them) is easier to read.
          • The <P> tags are unnecessary in javadocs. Some are also oddly formatted.
          • Some of the test javadocs still refer to BufferManager.java and related classes that have been removed
          Show
          chris.douglas Chris Douglas added a comment - Thanks for working through the build issues, and following up on the review feedback Vishwajeet Dusane . This may not require a branch. While the patch is large, much of it is removing code (to sever the WebHDFS dependency) and adding new tests. The former can be committed directly to trunk, and the latter we can commit in one or more followup issues (as subtasks of HADOOP-13257 ?). Any objections to this approach? A few minor comments from skimming the latest patch: AccessTokenProviderCache uses an unusual singleton pattern. Does it need to create a private instance of itself, rather than initializing a static map directly? AdlPermission#equals and #hashcode implementations are identical to what they override. Instead of reading from the config to report getAclBit (which is expensive, and unstable) it could either store the boolean or return basic FsPermission instances when disabled (and conversely, AdlPermission can always return true when the config knob is set). That this is a configurable boolean is surprising, and could be explained in this class's (empty) javadoc. TokenProviderType::fromString is unused In initialize, accountFQDN init can move up, so the accessTokenProvider init can be contiguous Both the subclasses of AzureADTokenProvider throw exceptions for all abstract methods save init, and appear not to use the private instance they configure. The init pattern here is pretty complicated, with initialize creating an instance and adding it to a global cache, and not using the reference it retains. Can this be simplified? If the instance were returned, AccessTokenProviderCache additions can be moved to AdlFileSystem , so all the get/add calls are in one class. The TokenProviderCache type seems unnecessary, if AzureADTP instances can return null to prevent caching (instead of the instanceof/cast) It's a matter of taste, but importing all the static members of AdlConfKeys in AdlFileSystem (or none of them) is easier to read. The <P> tags are unnecessary in javadocs. Some are also oddly formatted. Some of the test javadocs still refer to BufferManager.java and related classes that have been removed
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks for the heads up Chris Nauroth.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks for the heads up Chris Nauroth .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks for the detail review Chris Douglas. I have incorporated the code review comments. and removed the newly added live test cases.

          I will raise separate patch on HADOOP-13257 with newly live test cases and update.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks for the detail review Chris Douglas . I have incorporated the code review comments. and removed the newly added live test cases. I will raise separate patch on HADOOP-13257 with newly live test cases and update.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 34 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 59s trunk passed
          +1 compile 7m 35s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 57s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 7m 12s the patch passed
          +1 javac 7m 12s the patch passed
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 findbugs 2m 28s the patch passed
          +1 javadoc 1m 8s the patch passed
          +1 unit 8m 14s hadoop-common in the patch passed.
          +1 unit 3m 31s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          72m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837080/HADOOP-13037-004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 7223cfa66a4b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 69dd5fa
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10989/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10989/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 34 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 59s trunk passed +1 compile 7m 35s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 57s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 7m 12s the patch passed +1 javac 7m 12s the patch passed +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 findbugs 2m 28s the patch passed +1 javadoc 1m 8s the patch passed +1 unit 8m 14s hadoop-common in the patch passed. +1 unit 3m 31s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 72m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837080/HADOOP-13037-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 7223cfa66a4b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 69dd5fa Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10989/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10989/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks, Vishwajeet Dusane. I need to test it using some valid credentials, but this looks good, overall. I made a few minor changes as I was going through the patch, posted as v005. Please review.

          • Smartquote in index.md replaced with quote
          • Changed some uses of Configuration#get/setClass to improve type safety
          • Removed @VisibleForTesting annotations from AdlFileSystem fields
          • Removed deleted config params from TestConcurrentDataReadOperations
          • Collapsed static imports from AdlConfKeys
          • Removed a call to incrementReadOps from AdlFileSystem#getFileBlockLocations, as it's incremented in getFileStatus
          • Changed some unit tests that succeed for thrown exceptions to use the @Test(expected=ExceptionType.class) syntax
          • Flipped assertEquals(expected, actual) where these were inverted

          BTW- adls.xml adds the following:

          +  <property>
          +    <name>fs.contract.is-blobstore</name>
          +    <value>true</value>
          +  </property>
          

          I'm not familiar with this parameter. What is it controlling?

          Show
          chris.douglas Chris Douglas added a comment - Thanks, Vishwajeet Dusane . I need to test it using some valid credentials, but this looks good, overall. I made a few minor changes as I was going through the patch, posted as v005. Please review. Smartquote in index.md replaced with quote Changed some uses of Configuration#get/setClass to improve type safety Removed @VisibleForTesting annotations from AdlFileSystem fields Removed deleted config params from TestConcurrentDataReadOperations Collapsed static imports from AdlConfKeys Removed a call to incrementReadOps from AdlFileSystem#getFileBlockLocations , as it's incremented in getFileStatus Changed some unit tests that succeed for thrown exceptions to use the @Test(expected=ExceptionType.class) syntax Flipped assertEquals(expected, actual) where these were inverted BTW- adls.xml adds the following: + <property> + <name>fs.contract.is-blobstore</name> + <value>true</value> + </property> I'm not familiar with this parameter. What is it controlling?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm not familiar with this parameter.

          fs.contract.is-blobstore tells the tests that this an object store and expect consistency to be unpredictable. In HADOOP-13502 Chris Nauroth split up the marker to be more specific on the expected behaviour an FS can declare; this property is still the main one

          Show
          stevel@apache.org Steve Loughran added a comment - I'm not familiar with this parameter. fs.contract.is-blobstore tells the tests that this an object store and expect consistency to be unpredictable. In HADOOP-13502 Chris Nauroth split up the marker to be more specific on the expected behaviour an FS can declare; this property is still the main one
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 34 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 10m 40s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          +1 findbugs 2m 5s trunk passed
          +1 javadoc 1m 19s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 9m 45s the patch passed
          +1 javac 9m 45s the patch passed
          +1 checkstyle 1m 40s the patch passed
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 1m 25s the patch passed
          +1 unit 8m 27s hadoop-common in the patch passed.
          +1 unit 3m 41s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 46s The patch does not generate ASF License warnings.
          81m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837722/HADOOP-13037.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c8bd2b9f1ecd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ca33bdd
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11004/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11004/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 34 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 10m 40s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 47s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 1m 19s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 9m 45s the patch passed +1 javac 9m 45s the patch passed +1 checkstyle 1m 40s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 findbugs 2m 25s the patch passed +1 javadoc 1m 25s the patch passed +1 unit 8m 27s hadoop-common in the patch passed. +1 unit 3m 41s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 46s The patch does not generate ASF License warnings. 81m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837722/HADOOP-13037.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c8bd2b9f1ecd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ca33bdd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11004/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11004/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Chris Douglas and Vishwajeet Dusane, I advise setting fs.contract.is-blobstore to false. This flag is now strictly informational and does not control behavior of any tests. Since ADL is not a file system tree mapped onto a flat object store, it makes sense to set it to false.

          HADOOP-13502 introduced two new flags to control test behavior previously controlled by fs.contract.is-blobstore. For ADL, I believe the correct match to its semantics would be fs.contract.create-overwrites-directory=false and fs.contract.create-visibility-delayed=false. The latter is certainly important to support HBase expectations, and I know you want ADL to be able to support HBase. Running the ADL subclass of AbstractContractCreateTest would demonstrate if ADL is successfully implementing these semantics.

          FYI, I am about one week away from having a viable build environment, so I'll be delayed on testing the rebased patch that I mentioned. HADOOP-13687 likely will get committed without moving ADL in the source tree. We can likely accomplish that move within the scope of this JIRA instead.

          Show
          cnauroth Chris Nauroth added a comment - Chris Douglas and Vishwajeet Dusane , I advise setting fs.contract.is-blobstore to false. This flag is now strictly informational and does not control behavior of any tests. Since ADL is not a file system tree mapped onto a flat object store, it makes sense to set it to false. HADOOP-13502 introduced two new flags to control test behavior previously controlled by fs.contract.is-blobstore. For ADL, I believe the correct match to its semantics would be fs.contract.create-overwrites-directory=false and fs.contract.create-visibility-delayed=false. The latter is certainly important to support HBase expectations, and I know you want ADL to be able to support HBase. Running the ADL subclass of AbstractContractCreateTest would demonstrate if ADL is successfully implementing these semantics. FYI, I am about one week away from having a viable build environment, so I'll be delayed on testing the rebased patch that I mentioned. HADOOP-13687 likely will get committed without moving ADL in the source tree. We can likely accomplish that move within the scope of this JIRA instead.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks a lot Chris Douglas for incorporating and uploading v005 patch. i reviewed the changes you made and sounds good. +1 for the change set on v005.

          Also Chris Douglas and Chris Nauroth for highlighting fs.contract.is-blobstore configuration. As rightly mentioned by Chris Nauroth - fs.contract.is-blobstore is irrelevant and not required to be configured. I will update patch to remove fs.contract.is-blobstore configuration from adls.xml.

          Chris Nauroth and Steve Loughran - +1 on your suggestion on HADOOP-13687.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks a lot Chris Douglas for incorporating and uploading v005 patch. i reviewed the changes you made and sounds good. +1 for the change set on v005. Also Chris Douglas and Chris Nauroth for highlighting fs.contract.is-blobstore configuration. As rightly mentioned by Chris Nauroth - fs.contract.is-blobstore is irrelevant and not required to be configured. I will update patch to remove fs.contract.is-blobstore configuration from adls.xml . Chris Nauroth and Steve Loughran - +1 on your suggestion on HADOOP-13687 .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Removed fs.contract.is-blobstore config from adls.xml.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Removed fs.contract.is-blobstore config from adls.xml .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 34 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 8m 49s trunk passed
          +1 compile 12m 37s trunk passed
          +1 checkstyle 1m 42s trunk passed
          +1 mvnsite 1m 48s trunk passed
          +1 mvneclipse 1m 5s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 33s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 9m 39s the patch passed
          +1 javac 9m 39s the patch passed
          +1 checkstyle 1m 42s the patch passed
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 12s The patch has no ill-formed XML file.
          +1 findbugs 7m 24s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 7m 51s hadoop-common in the patch passed.
          +1 unit 3m 43s hadoop-azure-datalake in the patch passed.
          +1 asflicense 0m 47s The patch does not generate ASF License warnings.
          91m 24s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13037
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837834/HADOOP-13037.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux d6844b728fd4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / de3b4aa
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11013/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11013/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 34 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 8m 49s trunk passed +1 compile 12m 37s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 1m 5s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 9m 39s the patch passed +1 javac 9m 39s the patch passed +1 checkstyle 1m 42s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 12s The patch has no ill-formed XML file. +1 findbugs 7m 24s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 7m 51s hadoop-common in the patch passed. +1 unit 3m 43s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 91m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13037 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837834/HADOOP-13037.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux d6844b728fd4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de3b4aa Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11013/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11013/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks, everyone. I wasn't sure if this was set for a handful of cases (e.g., not updating timestamps). ADLS shouldn't behave like a blobstore, generally.

          I'll be delayed on testing the rebased patch that I mentioned. HADOOP-13687 likely will get committed without moving ADL in the source tree

          Makes sense. We can move ADLS after updating the contract tests.

          Show
          chris.douglas Chris Douglas added a comment - Thanks, everyone. I wasn't sure if this was set for a handful of cases (e.g., not updating timestamps). ADLS shouldn't behave like a blobstore, generally. I'll be delayed on testing the rebased patch that I mentioned. HADOOP-13687 likely will get committed without moving ADL in the source tree Makes sense. We can move ADLS after updating the contract tests.
          Hide
          chris.douglas Chris Douglas added a comment -

          I tested some basic shell operations against ADLS using v006 and ClientCredential auth. I'll commit this in a day or two.

          Aaron Fabbri, Chris Nauroth, Lei (Eddy) Xu, Steve Loughran: this seems to address the followup feedback from HADOOP-12666, particularly removing the WebHDFS dependency. Do you want to verify this before it's committed?

          Show
          chris.douglas Chris Douglas added a comment - I tested some basic shell operations against ADLS using v006 and ClientCredential auth. I'll commit this in a day or two. Aaron Fabbri , Chris Nauroth , Lei (Eddy) Xu , Steve Loughran : this seems to address the followup feedback from HADOOP-12666 , particularly removing the WebHDFS dependency. Do you want to verify this before it's committed?
          Hide
          fabbri Aaron Fabbri added a comment -

          Thanks for the ping Chris Douglas. Glad to see the followup on separating this from WebHDFS.

          Did you enable the FS contract tests and run those? Any issues?

          I notice this code has a separate way to disable the tests here:

          org.junit.Assume
          .assumeTrue(AdlStorageConfiguration.isContractTestEnabled());

          Show
          fabbri Aaron Fabbri added a comment - Thanks for the ping Chris Douglas . Glad to see the followup on separating this from WebHDFS. Did you enable the FS contract tests and run those? Any issues? I notice this code has a separate way to disable the tests here: org.junit.Assume .assumeTrue(AdlStorageConfiguration.isContractTestEnabled());
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Aaron Fabbri - For contract test related changes and comments from Steve Loughran and Chris Nauroth captured in HADOOP-13257. I will be raising a separate patch for contract test related optimization and correction as proposed by Chris Douglas under HADOOP-13257 task once HADOOP-13037 is committed.

          HADOOP-13257 would cover org.junit.Assume.assumeTrue(AdlStorageConfiguration.isContractTestEnabled()); like usage optimizations.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Aaron Fabbri - For contract test related changes and comments from Steve Loughran and Chris Nauroth captured in HADOOP-13257 . I will be raising a separate patch for contract test related optimization and correction as proposed by Chris Douglas under HADOOP-13257 task once HADOOP-13037 is committed. HADOOP-13257 would cover org.junit.Assume.assumeTrue(AdlStorageConfiguration.isContractTestEnabled()); like usage optimizations.
          Hide
          chris.douglas Chris Douglas added a comment -

          +1 on v006

          If there's nothing else on this iteration, I'll commit this in the morning (PST). It's a large refactor, but it's addressing existing feedback. HADOOP-13257 will focus on the contract tests (see v003), which merit specific review.

          Show
          chris.douglas Chris Douglas added a comment - +1 on v006 If there's nothing else on this iteration, I'll commit this in the morning (PST). It's a large refactor, but it's addressing existing feedback. HADOOP-13257 will focus on the contract tests (see v003), which merit specific review.
          Hide
          chris.douglas Chris Douglas added a comment -

          I committed this. Thanks, Vishwajeet.

          This is targeting branch-2, but we should complete the contract tests before we backport it.

          Show
          chris.douglas Chris Douglas added a comment - I committed this. Thanks, Vishwajeet. This is targeting branch-2, but we should complete the contract tests before we backport it.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Douglas - Thanks a lot for pushing this through. Will raise a contract test related patch on HADOOP-13257 over the weekend.

          Chris Nauroth and Steve Loughran Thanks a lot for the heads up on HADOOP-13687, will move ADLS once HADOOP-13257 is through.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Douglas - Thanks a lot for pushing this through. Will raise a contract test related patch on HADOOP-13257 over the weekend. Chris Nauroth and Steve Loughran Thanks a lot for the heads up on HADOOP-13687 , will move ADLS once HADOOP-13257 is through.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10822 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10822/)
          HADOOP-13037. Refactor Azure Data Lake Store as an independent (cdouglas: rev 5c61ad24887f76dfc5a5935b2c5dceb6bfd99417)

          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/package-info.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlFileSystemContractLive.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/TestDataForRead.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConcurrentDataReadOperations.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/oauth2/TestCachedRefreshTokenBasedAccessTokenProvider.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlPermission.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestableAdlFileSystem.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/Adl.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/CreateADLNoRedirectParam.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/BufferManager.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/package-info.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/AzureADClientCredentialBasedAccesTokenProvider.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/resources/META-INF/org.apache.hadoop.fs.FileSystem
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFsInputStream.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLPutOpParam.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/TokenProviderType.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLPostOpParam.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/resources/contract-test-options.xml
          • (edit) hadoop-tools/hadoop-azure-datalake/pom.xml
          • (delete) hadoop-tools/hadoop-azure-datalake/dev-support/findbugs-exclude.xml
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/resources/adls.xml
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/resources/log4j.properties
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/PrivateCachedRefreshTokenBasedAccessTokenProvider.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConfigurationSetting.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/ExpectedResponse.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlReadLive.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ReadADLNoRedirectParam.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/package-info.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/ADLConfKeys.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/package-info.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextCreateMkdirLive.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/CustomMockTokenProvider.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLVersionInfo.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestConcurrentDataReadOperations.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/Parallelized.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/ExpectedResponse.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/AzureADTokenProvider.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestACLFeatures.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/package-info.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/AdlMockWebServer.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestListStatus.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageContract.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAdlRead.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageConfiguration.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/CachedRefreshTokenBasedAccessTokenProvider.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFsOutputStream.java
          • (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/SdkTokenProviderAdapter.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/TestDataForRead.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextMainOperationsLive.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/AppendADLNoRedirectParam.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLGetOpParam.java
          • (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlDifferentSizeWritesLive.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestSplitSizeCalculation.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestAdlRead.java
          • (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/LeaseParam.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10822 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10822/ ) HADOOP-13037 . Refactor Azure Data Lake Store as an independent (cdouglas: rev 5c61ad24887f76dfc5a5935b2c5dceb6bfd99417) (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/package-info.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlFileSystemContractLive.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/TestDataForRead.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConcurrentDataReadOperations.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/oauth2/TestCachedRefreshTokenBasedAccessTokenProvider.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlPermission.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestableAdlFileSystem.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/Adl.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/CreateADLNoRedirectParam.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/BufferManager.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/package-info.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/AzureADClientCredentialBasedAccesTokenProvider.java (add) hadoop-tools/hadoop-azure-datalake/src/main/resources/META-INF/org.apache.hadoop.fs.FileSystem (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFsInputStream.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLPutOpParam.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAzureADTokenProvider.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/TokenProviderType.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLPostOpParam.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestRelativePathFormation.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/resources/contract-test-options.xml (edit) hadoop-tools/hadoop-azure-datalake/pom.xml (delete) hadoop-tools/hadoop-azure-datalake/dev-support/findbugs-exclude.xml (edit) hadoop-tools/hadoop-azure-datalake/src/test/resources/adls.xml (add) hadoop-tools/hadoop-azure-datalake/src/test/resources/log4j.properties (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/PrivateCachedRefreshTokenBasedAccessTokenProvider.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConfigurationSetting.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/ExpectedResponse.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlReadLive.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ReadADLNoRedirectParam.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/package-info.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/ADLConfKeys.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/oauth2/package-info.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextCreateMkdirLive.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/CustomMockTokenProvider.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLVersionInfo.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestConcurrentDataReadOperations.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/Parallelized.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/ExpectedResponse.java (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/AzureADTokenProvider.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestACLFeatures.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/package-info.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/AdlMockWebServer.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestListStatus.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageContract.java (add) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAdlRead.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageConfiguration.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/oauth2/CachedRefreshTokenBasedAccessTokenProvider.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFsOutputStream.java (add) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/SdkTokenProviderAdapter.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/TestDataForRead.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextMainOperationsLive.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/AppendADLNoRedirectParam.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLGetOpParam.java (edit) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlDifferentSizeWritesLive.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestSplitSizeCalculation.java (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestAdlRead.java (delete) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/LeaseParam.java
          Hide
          hitesh Hitesh Shah added a comment -

          Chris Douglas Chris Nauroth Vishwajeet Dusane Steve Loughran Given that HADOOP-13687 and HADOOP-13257 are resolved, can this be backported to branch-2?

          Show
          hitesh Hitesh Shah added a comment - Chris Douglas Chris Nauroth Vishwajeet Dusane Steve Loughran Given that HADOOP-13687 and HADOOP-13257 are resolved, can this be backported to branch-2?
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for backporting to branch-2.

          Show
          cnauroth Chris Nauroth added a comment - +1 for backporting to branch-2.
          Hide
          chris.douglas Chris Douglas added a comment -

          Given that HADOOP-13687 and HADOOP-13257 are resolved, can this be backported to branch-2?

          +1 Removing the subclassing was the condition for backporting to branch-2.

          Show
          chris.douglas Chris Douglas added a comment - Given that HADOOP-13687 and HADOOP-13257 are resolved, can this be backported to branch-2? +1 Removing the subclassing was the condition for backporting to branch-2.
          Hide
          chris.douglas Chris Douglas added a comment -

          Reopening for backport to branch-2. Talked to Junping Du, and will backport to branch-2.8 if there are no objections.

          Show
          chris.douglas Chris Douglas added a comment - Reopening for backport to branch-2. Talked to Junping Du , and will backport to branch-2.8 if there are no objections.
          Hide
          chris.douglas Chris Douglas added a comment -

          Committed through branch-2.8.0

          Show
          chris.douglas Chris Douglas added a comment - Committed through branch-2.8.0

            People

            • Assignee:
              vishwajeet.dusane Vishwajeet Dusane
              Reporter:
              shrikant Shrikant Naidu
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development