Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12666

Support Microsoft Azure Data Lake - as a file system in Hadoop

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: fs/adl, tools
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hadoop now supports integration with Azure Data Lake as an alternative Hadoop-compatible file system. Please refer to the Hadoop site documentation of Azure Data Lake for details on usage and configuration.

      Description

      Description

      This JIRA describes a new file system implementation for accessing Microsoft Azure Data Lake Store (ADL) from within Hadoop. This would enable existing Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as input or output.

      ADL is ultra-high capacity, Optimized for massive throughput with rich management and security features. More details available at https://azure.microsoft.com/en-us/services/data-lake-store/

      1. Create_Read_Hadoop_Adl_Store_Semantics.pdf
        431 kB
        Vishwajeet Dusane
      2. HADOOP-12666-002.patch
        218 kB
        Vishwajeet Dusane
      3. HADOOP-12666-003.patch
        220 kB
        Vishwajeet Dusane
      4. HADOOP-12666-004.patch
        220 kB
        Vishwajeet Dusane
      5. HADOOP-12666-005.patch
        220 kB
        Vishwajeet Dusane
      6. HADOOP-12666-006.patch
        233 kB
        Vishwajeet Dusane
      7. HADOOP-12666-007.patch
        107 kB
        Vishwajeet Dusane
      8. HADOOP-12666-008.patch
        108 kB
        Vishwajeet Dusane
      9. HADOOP-12666-009.patch
        108 kB
        Vishwajeet Dusane
      10. HADOOP-12666-010.patch
        106 kB
        Vishwajeet Dusane
      11. HADOOP-12666-011.patch
        104 kB
        Vishwajeet Dusane
      12. HADOOP-12666-012.patch
        110 kB
        Chris Douglas
      13. HADOOP-12666-013.patch
        110 kB
        Chris Nauroth
      14. HADOOP-12666-014.patch
        111 kB
        Chris Douglas
      15. HADOOP-12666-015.patch
        111 kB
        Vishwajeet Dusane
      16. HADOOP-12666-016.patch
        112 kB
        Vishwajeet Dusane
      17. HADOOP-12666-1.patch
        179 kB
        Vishwajeet Dusane

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 6m 31s trunk passed
          +1 compile 1m 0s trunk passed with JDK v1.8.0_66
          +1 compile 1m 13s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools
          +1 findbugs 0m 1s trunk passed
          +1 javadoc 0m 44s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 57s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 2m 34s the patch passed
          +1 compile 1m 2s the patch passed with JDK v1.8.0_66
          -1 javac 3m 56s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149)
          +1 javac 1m 2s the patch passed
          +1 compile 1m 16s the patch passed with JDK v1.7.0_91
          -1 javac 5m 13s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new + 148 unchanged - 0 fixed = 150 total (was 148)
          +1 javac 1m 16s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 3m 39s the patch passed
          +1 mvneclipse 0m 51s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools
          +1 findbugs 0m 0s the patch passed
          -1 javadoc 0m 47s hadoop-tools in the patch failed with JDK v1.8.0_66.
          -1 javadoc 0m 47s hadoop-tools in the patch failed with JDK v1.8.0_66.
          -1 javadoc 4m 0s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 20 new + 10 unchanged - 0 fixed = 30 total (was 10)
          -1 javadoc 4m 0s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 20 new + 10 unchanged - 0 fixed = 30 total (was 10)
          +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91
          +1 unit 30m 46s hadoop-tools in the patch passed with JDK v1.8.0_66.
          +1 unit 30m 52s hadoop-tools in the patch passed with JDK v1.8.0_66.
          -1 unit 31m 52s hadoop-tools in the patch failed with JDK v1.7.0_91.
          +1 unit 32m 28s hadoop-tools in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          153m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785183/HADOOP-12666-002.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux 9d29136e72c0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c9a09d6
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          javac hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-compile-javac-hadoop-tools-jdk1.8.0_66.txt
          javac hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-compile-javac-hadoop-tools-jdk1.7.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_66.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_66.txt
          javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
          javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/testReport/
          modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 6m 31s trunk passed +1 compile 1m 0s trunk passed with JDK v1.8.0_66 +1 compile 1m 13s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 20s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools +1 findbugs 0m 1s trunk passed +1 javadoc 0m 44s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 57s trunk passed with JDK v1.7.0_91 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 2m 34s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_66 -1 javac 3m 56s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149) +1 javac 1m 2s the patch passed +1 compile 1m 16s the patch passed with JDK v1.7.0_91 -1 javac 5m 13s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new + 148 unchanged - 0 fixed = 150 total (was 148) +1 javac 1m 16s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 3m 39s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools +1 findbugs 0m 0s the patch passed -1 javadoc 0m 47s hadoop-tools in the patch failed with JDK v1.8.0_66. -1 javadoc 0m 47s hadoop-tools in the patch failed with JDK v1.8.0_66. -1 javadoc 4m 0s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 20 new + 10 unchanged - 0 fixed = 30 total (was 10) -1 javadoc 4m 0s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 20 new + 10 unchanged - 0 fixed = 30 total (was 10) +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91 +1 unit 30m 46s hadoop-tools in the patch passed with JDK v1.8.0_66. +1 unit 30m 52s hadoop-tools in the patch passed with JDK v1.8.0_66. -1 unit 31m 52s hadoop-tools in the patch failed with JDK v1.7.0_91. +1 unit 32m 28s hadoop-tools in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 153m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785183/HADOOP-12666-002.patch JIRA Issue HADOOP-12666 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 9d29136e72c0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c9a09d6 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 javac hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-compile-javac-hadoop-tools-jdk1.8.0_66.txt javac hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-compile-javac-hadoop-tools-jdk1.7.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_66.txt javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/testReport/ modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8493/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 1m 9s trunk passed with JDK v1.8.0_66
          +1 compile 1m 18s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 50s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 0m 48s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 58s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 35s the patch passed
          +1 compile 1m 9s the patch passed with JDK v1.8.0_66
          +1 javac 1m 9s the patch passed
          +1 compile 1m 17s the patch passed with JDK v1.7.0_91
          +1 javac 1m 17s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 3m 40s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools
          +1 findbugs 0m 0s the patch passed
          -1 javadoc 1m 58s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403)
          -1 javadoc 1m 59s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403)
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66
          -1 javadoc 4m 20s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10)
          -1 javadoc 4m 20s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10)
          +1 javadoc 2m 6s the patch passed with JDK v1.7.0_91
          +1 unit 31m 24s hadoop-tools in the patch passed with JDK v1.8.0_66.
          +1 unit 30m 28s hadoop-tools in the patch passed with JDK v1.8.0_66.
          +1 unit 32m 4s hadoop-tools in the patch passed with JDK v1.7.0_91.
          +1 unit 33m 17s hadoop-tools in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          156m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785489/HADOOP-12666-003.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux f37c2d652e96 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8f2622b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          javadoc hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt
          javadoc hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt
          javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
          javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/testReport/
          modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 1m 9s trunk passed with JDK v1.8.0_66 +1 compile 1m 18s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 50s trunk passed +1 mvneclipse 0m 33s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools +1 findbugs 0m 0s trunk passed +1 javadoc 0m 48s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 58s trunk passed with JDK v1.7.0_91 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 35s the patch passed +1 compile 1m 9s the patch passed with JDK v1.8.0_66 +1 javac 1m 9s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_91 +1 javac 1m 17s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 3m 40s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools +1 findbugs 0m 0s the patch passed -1 javadoc 1m 58s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403) -1 javadoc 1m 59s hadoop-tools-jdk1.8.0_66 with JDK v1.8.0_66 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403) +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66 -1 javadoc 4m 20s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10) -1 javadoc 4m 20s hadoop-tools-jdk1.7.0_91 with JDK v1.7.0_91 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10) +1 javadoc 2m 6s the patch passed with JDK v1.7.0_91 +1 unit 31m 24s hadoop-tools in the patch passed with JDK v1.8.0_66. +1 unit 30m 28s hadoop-tools in the patch passed with JDK v1.8.0_66. +1 unit 32m 4s hadoop-tools in the patch passed with JDK v1.7.0_91. +1 unit 33m 17s hadoop-tools in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 156m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785489/HADOOP-12666-003.patch JIRA Issue HADOOP-12666 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux f37c2d652e96 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8f2622b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 javadoc hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt javadoc hadoop-tools-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_66.txt javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt javadoc hadoop-tools-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/testReport/ modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8503/console This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Not seeing in the report any Javadoc warning related to this patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Not seeing in the report any Javadoc warning related to this patch.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Resubmitting patch for automated jenkins build. Javadoc issues reported in earlier build was in different component and not caused due to this patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Resubmitting patch for automated jenkins build. Javadoc issues reported in earlier build was in different component and not caused due to this patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 32s trunk passed
          +1 compile 0m 59s trunk passed with JDK v1.8.0_72
          +1 compile 1m 14s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 0m 45s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 57s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 35s the patch passed
          +1 compile 1m 2s the patch passed with JDK v1.8.0_72
          +1 javac 1m 2s the patch passed
          +1 compile 1m 17s the patch passed with JDK v1.7.0_95
          +1 javac 1m 17s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 3m 39s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools
          +1 findbugs 0m 0s the patch passed
          -1 javadoc 1m 49s hadoop-tools-jdk1.8.0_72 with JDK v1.8.0_72 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403)
          -1 javadoc 1m 50s hadoop-tools-jdk1.8.0_72 with JDK v1.8.0_72 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403)
          +1 javadoc 1m 33s the patch passed with JDK v1.8.0_72
          -1 javadoc 4m 4s hadoop-tools-jdk1.7.0_95 with JDK v1.7.0_95 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10)
          -1 javadoc 4m 4s hadoop-tools-jdk1.7.0_95 with JDK v1.7.0_95 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10)
          +1 javadoc 1m 58s the patch passed with JDK v1.7.0_95
          +1 unit 30m 59s hadoop-tools in the patch passed with JDK v1.8.0_72.
          +1 unit 32m 11s hadoop-tools in the patch passed with JDK v1.8.0_72.
          +1 unit 32m 27s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 32m 33s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          156m 5s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786063/HADOOP-12666-004.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux 7c70514ca3f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fa328e2
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          javadoc hadoop-tools-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_72.txt
          javadoc hadoop-tools-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_72.txt
          javadoc hadoop-tools-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_95.txt
          javadoc hadoop-tools-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/testReport/
          modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 32s trunk passed +1 compile 0m 59s trunk passed with JDK v1.8.0_72 +1 compile 1m 14s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 20s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools +1 findbugs 0m 0s trunk passed +1 javadoc 0m 45s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 57s trunk passed with JDK v1.7.0_95 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 35s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_72 +1 javac 1m 2s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_95 +1 javac 1m 17s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 3m 39s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools hadoop-tools +1 findbugs 0m 0s the patch passed -1 javadoc 1m 49s hadoop-tools-jdk1.8.0_72 with JDK v1.8.0_72 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403) -1 javadoc 1m 50s hadoop-tools-jdk1.8.0_72 with JDK v1.8.0_72 generated 403 new + 403 unchanged - 0 fixed = 806 total (was 403) +1 javadoc 1m 33s the patch passed with JDK v1.8.0_72 -1 javadoc 4m 4s hadoop-tools-jdk1.7.0_95 with JDK v1.7.0_95 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10) -1 javadoc 4m 4s hadoop-tools-jdk1.7.0_95 with JDK v1.7.0_95 generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10) +1 javadoc 1m 58s the patch passed with JDK v1.7.0_95 +1 unit 30m 59s hadoop-tools in the patch passed with JDK v1.8.0_72. +1 unit 32m 11s hadoop-tools in the patch passed with JDK v1.8.0_72. +1 unit 32m 27s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 32m 33s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 156m 5s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786063/HADOOP-12666-004.patch JIRA Issue HADOOP-12666 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 7c70514ca3f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fa328e2 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 javadoc hadoop-tools-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_72.txt javadoc hadoop-tools-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.8.0_72.txt javadoc hadoop-tools-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_95.txt javadoc hadoop-tools-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/testReport/ modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8516/console This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          Hi. First question: Why is there a separate checkstyle.xml for this module? Shouldn't it just use the top level one?

          Show
          fabbri Aaron Fabbri added a comment - Hi. First question: Why is there a separate checkstyle.xml for this module? Shouldn't it just use the top level one?
          Hide
          cnauroth Chris Nauroth added a comment -

          Why is there a separate checkstyle.xml for this module?

          Judging from the content, I am guessing it was copied from hadoop-tools/hadoop-azure/src/config/checkstyle.xml. However, the checkstyle.xml in hadoop-azure is really a historical artifact that we intend to clean up, not a pattern meant to propagate into new code.

          HADOOP-11899 tracks the cleanup for hadoop-azure. New code contributions like this one can stick to the standard Hadoop style from the beginning.

          Show
          cnauroth Chris Nauroth added a comment - Why is there a separate checkstyle.xml for this module? Judging from the content, I am guessing it was copied from hadoop-tools/hadoop-azure/src/config/checkstyle.xml. However, the checkstyle.xml in hadoop-azure is really a historical artifact that we intend to clean up, not a pattern meant to propagate into new code. HADOOP-11899 tracks the cleanup for hadoop-azure. New code contributions like this one can stick to the standard Hadoop style from the beginning.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Nauroth is right. checkstyle.xml is derived from the hadoop-azure. Understood that it was historical, i have started working on removing custom checkstyle.xml dependency from the code.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Nauroth is right. checkstyle.xml is derived from the hadoop-azure. Understood that it was historical, i have started working on removing custom checkstyle.xml dependency from the code.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Description

          This JIRA describes a new file system implementation for accessing Microsoft Azure Data Lake Store (ADL) from within Hadoop. This would enable existing Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as input or output.

          ADL is ultra-high capacity, Optimized for massive throughput with rich management and security features. More details available at https://azure.microsoft.com/en-us/services/data-lake-store/

          High level design

          ADL file system exposes RESTful interfaces compatible with WebHdfs specification 2.7.1.
          At a high level, the code here extends the SWebHdfsFileSystem class to provide an implementation for accessing ADL storage; the scheme ADL is used for accessing it over HTTPS. We use the URI scheme:

          adl://<URI to account>/path/to/file

          to address individual Files/Folders. Tests are implemented mostly using a Contract implementation for the ADL functionality, with an option to test against a real ADL storage if configured.

          Credits and history

          This has been ongoing work for a while, and the early version of this work can be seen in. Credit for this work goes to the team: Vishwajeet Dusane, Santhosh G Nayak, Shradha Revankar, kiran, Chakrapani Bhat, Omkar Aradhya K S, Sneha Vijayarajan, Sai Prasanna jitendra

          Test

          Besides Contract tests, we have used ADL as the additional file system in the current public preview release. Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Description This JIRA describes a new file system implementation for accessing Microsoft Azure Data Lake Store (ADL) from within Hadoop. This would enable existing Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as input or output. ADL is ultra-high capacity, Optimized for massive throughput with rich management and security features. More details available at https://azure.microsoft.com/en-us/services/data-lake-store/ High level design ADL file system exposes RESTful interfaces compatible with WebHdfs specification 2.7.1. At a high level, the code here extends the SWebHdfsFileSystem class to provide an implementation for accessing ADL storage; the scheme ADL is used for accessing it over HTTPS. We use the URI scheme: adl: //<URI to account>/path/to/file to address individual Files/Folders. Tests are implemented mostly using a Contract implementation for the ADL functionality, with an option to test against a real ADL storage if configured. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in. Credit for this work goes to the team: Vishwajeet Dusane , Santhosh G Nayak , Shradha Revankar , kiran , Chakrapani Bhat , Omkar Aradhya K S , Sneha Vijayarajan , Sai Prasanna jitendra Test Besides Contract tests, we have used ADL as the additional file system in the current public preview release. Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Incorporated review comment from Chris Douglas
          1. FileStatusCacheManger.java - Remove dependency from Timer to manage cache expiry time. Use LinkedHashMap with object insert time and expiry in seconds.
          2. FileStatusCacheManger.java - Cache to hold upto 5000 FileStatus instances
          3. ADLFeatureConfig.java - Removed global configuration to Webhdfds specific instance
          4. Added test cases to verify caching scenario
          5. PrivateAzureDataLakeFileSystem.java replaced "swebhdfs" scheme with "adl" to avoid wrong class being loaded for "swebhdfs" config
          6. Removed custom checkstyle.xml reference to default checkstyle.xml configuration

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Incorporated review comment from Chris Douglas 1. FileStatusCacheManger.java - Remove dependency from Timer to manage cache expiry time. Use LinkedHashMap with object insert time and expiry in seconds. 2. FileStatusCacheManger.java - Cache to hold upto 5000 FileStatus instances 3. ADLFeatureConfig.java - Removed global configuration to Webhdfds specific instance 4. Added test cases to verify caching scenario 5. PrivateAzureDataLakeFileSystem.java replaced "swebhdfs" scheme with "adl" to avoid wrong class being loaded for "swebhdfs" config 6. Removed custom checkstyle.xml reference to default checkstyle.xml configuration
          Hide
          fabbri Aaron Fabbri added a comment -
          +  /**
          +   * Constructor.
          +   */                                                                                                        
          +  public CachedRefreshTokenBasedAccessTokenProvider() {                                                      
          +    super();                                                                                                 
          +    if (instance == null) {
          +      instance = new ConfRefreshTokenBasedAccessTokenProvider();                                             
          +    }                                                                                                        
          +  }
          

          You can omit call to super() here.

          Same thing in PrivateDebugAzureDataLakeFileSystem()

          +package com.microsoft.azure.datalake.store;
          +
          +import org.apache.hadoop.hdfs.web.PrivateAzureDataLakeFileSystem;
          +
          +class AdlFileSystem extends PrivateAzureDataLakeFileSystem {
          +
          

          Why is PrivateAzureDataLakeFileSystem public?

          More importantly, shouldn't you move all the code into org.apache.hadoop.fs.azure? As is, it is spread between com.microsoft.azure, org.apache.hadoop.fs.azure, and org.apache.hadoop.hdfs.web.

          +package org.apache.hadoop.hdfs.web;
          +
          +/**
          + * Constants.
          + */
          +public final class ADLConfKeys {
          +  public static final String
          +      ADL_FEATURE_CONCURRENT_READ_AHEAD_MAX_CONCURRENT_CONN =
          +      "ADL.Feature.Override.ReadAhead.MAX.Concurrent.Connection";
          +  public static final int
          +      ADL_FEATURE_CONCURRENT_READ_AHEAD_MAX_CONCURRENT_CONN_DEFAULT = 2;
          +  public static final String ADL_EVENTS_TRACKING_SOURCE =
          +      "adl.events.tracking.source";
          +  public static final String ADL_EVENTS_TRACKING_CLUSTERNAME =
          +      "adl.events.tracking.clustername";
          +  public static final String ADL_TRACKING_JOB_ID = "adl.tracking.job.id";
          

          Please be consistent with all lowercase config names, and document them in

          {core-default.xml}

          .

          Need to run.. more comments later.

          Show
          fabbri Aaron Fabbri added a comment - + /** + * Constructor. + */ + public CachedRefreshTokenBasedAccessTokenProvider() { + super (); + if (instance == null ) { + instance = new ConfRefreshTokenBasedAccessTokenProvider(); + } + } You can omit call to super() here. Same thing in PrivateDebugAzureDataLakeFileSystem() + package com.microsoft.azure.datalake.store; + + import org.apache.hadoop.hdfs.web.PrivateAzureDataLakeFileSystem; + +class AdlFileSystem extends PrivateAzureDataLakeFileSystem { + Why is PrivateAzureDataLakeFileSystem public? More importantly, shouldn't you move all the code into org.apache.hadoop.fs.azure? As is, it is spread between com.microsoft.azure , org.apache.hadoop.fs.azure , and org.apache.hadoop.hdfs.web . + package org.apache.hadoop.hdfs.web; + +/** + * Constants. + */ + public final class ADLConfKeys { + public static final String + ADL_FEATURE_CONCURRENT_READ_AHEAD_MAX_CONCURRENT_CONN = + "ADL.Feature.Override.ReadAhead.MAX.Concurrent.Connection" ; + public static final int + ADL_FEATURE_CONCURRENT_READ_AHEAD_MAX_CONCURRENT_CONN_DEFAULT = 2; + public static final String ADL_EVENTS_TRACKING_SOURCE = + "adl.events.tracking.source" ; + public static final String ADL_EVENTS_TRACKING_CLUSTERNAME = + "adl.events.tracking.clustername" ; + public static final String ADL_TRACKING_JOB_ID = "adl.tracking.job.id" ; Please be consistent with all lowercase config names, and document them in {core-default.xml} . Need to run.. more comments later.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hey, Vishwajeet Dusane

          Thanks for working on this nice patch.

          Have a few questions,

          • FileStatusCacheManager.java
             * ACID properties are maintained in overloaded api in @see
             * PrivateAzureDataLakeFileSystem class.
            
          • You mentioned in the above comments. But PrivateAzureDataLakeFileSystem does not call it within synchronized calls (e.g., PrivateAzureDataLakeFileSystem#create. Although syncMap is a synchronizedMap, putFileStatus has multiple operations on syncMap, which can not guarantee atomicity.
          • It might be a better idea to provide atomicity in PrivateAzureDataLakeFileSystem. A couple of places have multiple cache calls within the same function (e.g., rename()).
          • It might be a good idea to rename FileStatusCacheManager#getFileStatus, putFileStatus, removeFileStatus to get/put/remove, because the class name already clearly indicates the context.
          • FileStatusCacheObject can only store an absolute expiration time. And its methods can be package-level methods.
          • I saw a few places, e.g., PrivateAzureDataLakeFileSystem#rename/delete, that clear the cache if the param is a directory. Could you justify the reason behind this? Would it cause noticeable performance degradation? Or as an alternative, using LinkedList + TreeMap for FileStatusCacheManager?
          • One general question, is this FileStatusCacheManager in HdfsClient? If it is the case, how do you make them consistent across clients on multiple nodes?
          • Similar to above question, could you provide a reference architecture of how to run a cluster on Azure Data Lake?
          •        if (b == null) {
                      throw new NullPointerException();
                    } else if (off < 0 || len < 0 || len > b.length - off) {
                      throw new IndexOutOfBoundsException();
                    } else if (len == 0) {
                      return 0;
                    }
            

          Can we use Precondtions here? It will be more descriptive.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hey, Vishwajeet Dusane Thanks for working on this nice patch. Have a few questions, FileStatusCacheManager.java * ACID properties are maintained in overloaded api in @see * PrivateAzureDataLakeFileSystem class. You mentioned in the above comments. But PrivateAzureDataLakeFileSystem does not call it within synchronized calls (e.g., PrivateAzureDataLakeFileSystem#create . Although syncMap is a synchronizedMap , putFileStatus has multiple operations on syncMap , which can not guarantee atomicity. It might be a better idea to provide atomicity in PrivateAzureDataLakeFileSystem . A couple of places have multiple cache calls within the same function (e.g., rename() ). It might be a good idea to rename FileStatusCacheManager#getFileStatus, putFileStatus, removeFileStatus to get/put/remove , because the class name already clearly indicates the context. FileStatusCacheObject can only store an absolute expiration time. And its methods can be package-level methods. I saw a few places, e.g., PrivateAzureDataLakeFileSystem#rename/delete , that clear the cache if the param is a directory. Could you justify the reason behind this? Would it cause noticeable performance degradation? Or as an alternative, using LinkedList + TreeMap for FileStatusCacheManager? One general question, is this FileStatusCacheManager in HdfsClient ? If it is the case, how do you make them consistent across clients on multiple nodes? Similar to above question, could you provide a reference architecture of how to run a cluster on Azure Data Lake? if (b == null ) { throw new NullPointerException(); } else if (off < 0 || len < 0 || len > b.length - off) { throw new IndexOutOfBoundsException(); } else if (len == 0) { return 0; } Can we use Precondtions here? It will be more descriptive.
          Hide
          cnauroth Chris Nauroth added a comment -

          Vishwajeet Dusane et al, thank you for the contribution. I'd like to start by discussing a few things that we would commonly ask of any contribution of a new Haodop-compatible file system:

          1. A new feature contribution requires end user documentation. For inspiration, you can take a look at the existing documentation for S3, Azure and Swift. The source for these pages is in the Hadoop code as Markdown, so you can look for *.md to find examples. It's good to discuss how the semantics of this file system will be the same as the semantics of HDFS and what will differ/work differently/not work at all. One particular point worth documenting is authentication, since there is a reliance on the WebHDFS OAuth feature here.
          2. Are there any tests that actually integrate with the back-end service, or is everything using a mock server right now? We've seen in the past that real integration testing is important. The mock-based testing is valuable too, because Apache Jenkins won't have the capability to run against the live service. A useful setup has been to have a common set of tests that support running in both "mock mode" and "live mode". For an example of this, see the hadoop-azure test suites.
          3. We have a set of file system contract tests that exercise the expected semantics of file systems. We ask that new file system implementations implement this contract test suite. This is also a great way to see how closely a file system matches the semantics of HDFS. hadoop-azure does not yet run these tests, but HADOOP-12535 has a proposed patch to start doing so.

          I scanned through the code. This is not a comprehensive review, but there are a few significant things that I've spotted so far:

          1. Some classes are using com.microsoft.azure.datalake as the root of the package name. All package names must use the standard org.apache.hadoop prefix instead.
          2. Can you please add JavaDoc comments at the top of PrivateAzureDataLakeFileSystem and ADLFileSystem to describe the design intent behind splitting the implementation classes like this? A potential source of confusion is that PrivateAzureDataLakeFileSystem is using the public Java access modifier, so the "private" in the class name is unrelated to Java language visibility semantics.
          3. CachedRefreshTokenBasedAccessTokenProvider: The constructor might end up creating multiple instances. There is no locking around creation of the shared instance.
          4. PrivateDebugAzureDataLakeFileSystem/PrivateDebugAzureDataLake: It is unusual to control level of logging by swapping out the FileSystem implementation. Users will expect that logging level is controlled the standard way through Log4J configuration. Can we eliminate these classes and move appropriate logging at debug and trace level directly into PrivateAzureDataLakeFileSystem?
          5. ADLLogger: With consideration of the above, I'm not sure that this class adds any value over directly calling an SLF4J Log instance from the relevant call sites. What do you think of dropping this class and converting call sites to call SLF4J directly?
          6. ADLConfKeys: There is an established precedent in Hadoop configuration for using all lower-case letters in configuration keys, delimited by '.' for separation of words. I recommend staying consistent with that pattern.
          7. You are treading new ground in choosing to cache FileStatus instances locally on the client side to achieve performance. This seems kind of like the Linux concept of the dentry cache. The tricky thing is cache coherency across mutliple processes running in the cluster and consistency semantics. I don't have a specific example of something that could break, but this seems like there could be a risk of something in the ecosystem breaking because of this. No other Hadoop-compatible file system works like this. This also relates back to my earlier point about documentation of expected semantics.
          8. ADLConfKeys#LOG_VERSION: I'm not sure what this is meant to be. Is the goal to pass the current build version number to the back end? If so, then consider looking at org.apache.hadoop.util.VersionInfo, which is already tied into the Apache release process. This version string doesn't need to be updated manually.
          9. TestAAAAAUploadBenchmark/TestAAAAADownloadBenchmark: From the comments, it sounds like the "AAAA" in these test suite names is intended to force a specific execution order, because some instability was seen while running the tests in a different order. This isn't guaranteed to work though, because Hadoop tests are gradually being converted to run in multiple concurrent processes for performance reasons. In that model, you could have multiple processes each running one of these "TestAAAA" suites, possibly also running other suites, and ordering is ultimately non-deterministic. If there is a test isolation bug, then please find root cause, fix the bug, and rename these test suites without the "AAAA".
          Show
          cnauroth Chris Nauroth added a comment - Vishwajeet Dusane et al, thank you for the contribution. I'd like to start by discussing a few things that we would commonly ask of any contribution of a new Haodop-compatible file system: A new feature contribution requires end user documentation. For inspiration, you can take a look at the existing documentation for S3 , Azure and Swift . The source for these pages is in the Hadoop code as Markdown, so you can look for *.md to find examples. It's good to discuss how the semantics of this file system will be the same as the semantics of HDFS and what will differ/work differently/not work at all. One particular point worth documenting is authentication, since there is a reliance on the WebHDFS OAuth feature here. Are there any tests that actually integrate with the back-end service, or is everything using a mock server right now? We've seen in the past that real integration testing is important. The mock-based testing is valuable too, because Apache Jenkins won't have the capability to run against the live service. A useful setup has been to have a common set of tests that support running in both "mock mode" and "live mode". For an example of this, see the hadoop-azure test suites. We have a set of file system contract tests that exercise the expected semantics of file systems. We ask that new file system implementations implement this contract test suite. This is also a great way to see how closely a file system matches the semantics of HDFS. hadoop-azure does not yet run these tests, but HADOOP-12535 has a proposed patch to start doing so. I scanned through the code. This is not a comprehensive review, but there are a few significant things that I've spotted so far: Some classes are using com.microsoft.azure.datalake as the root of the package name. All package names must use the standard org.apache.hadoop prefix instead. Can you please add JavaDoc comments at the top of PrivateAzureDataLakeFileSystem and ADLFileSystem to describe the design intent behind splitting the implementation classes like this? A potential source of confusion is that PrivateAzureDataLakeFileSystem is using the public Java access modifier, so the "private" in the class name is unrelated to Java language visibility semantics. CachedRefreshTokenBasedAccessTokenProvider : The constructor might end up creating multiple instances. There is no locking around creation of the shared instance . PrivateDebugAzureDataLakeFileSystem / PrivateDebugAzureDataLake : It is unusual to control level of logging by swapping out the FileSystem implementation. Users will expect that logging level is controlled the standard way through Log4J configuration. Can we eliminate these classes and move appropriate logging at debug and trace level directly into PrivateAzureDataLakeFileSystem ? ADLLogger : With consideration of the above, I'm not sure that this class adds any value over directly calling an SLF4J Log instance from the relevant call sites. What do you think of dropping this class and converting call sites to call SLF4J directly? ADLConfKeys : There is an established precedent in Hadoop configuration for using all lower-case letters in configuration keys, delimited by '.' for separation of words. I recommend staying consistent with that pattern. You are treading new ground in choosing to cache FileStatus instances locally on the client side to achieve performance. This seems kind of like the Linux concept of the dentry cache. The tricky thing is cache coherency across mutliple processes running in the cluster and consistency semantics. I don't have a specific example of something that could break, but this seems like there could be a risk of something in the ecosystem breaking because of this. No other Hadoop-compatible file system works like this. This also relates back to my earlier point about documentation of expected semantics. ADLConfKeys#LOG_VERSION : I'm not sure what this is meant to be. Is the goal to pass the current build version number to the back end? If so, then consider looking at org.apache.hadoop.util.VersionInfo , which is already tied into the Apache release process. This version string doesn't need to be updated manually. TestAAAAAUploadBenchmark / TestAAAAADownloadBenchmark : From the comments, it sounds like the "AAAA" in these test suite names is intended to force a specific execution order, because some instability was seen while running the tests in a different order. This isn't guaranteed to work though, because Hadoop tests are gradually being converted to run in multiple concurrent processes for performance reasons. In that model, you could have multiple processes each running one of these "TestAAAA" suites, possibly also running other suites, and ordering is ultimately non-deterministic. If there is a test isolation bug, then please find root cause, fix the bug, and rename these test suites without the "AAAA".
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          aaron fabbri Thanks for the comments. i agree with the comments.
          1. I removed super() invocation.
          2. PrivateAzureDataLakeFileSystem is public so that AdlFileSystem can extend in different namespace. We are having similar discussion on the namespace as you suggested. I will update the namespaces as well.
          3. Should i update in hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ? Since these configuration are specific to ADL file system only.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - aaron fabbri Thanks for the comments. i agree with the comments. 1. I removed super() invocation. 2. PrivateAzureDataLakeFileSystem is public so that AdlFileSystem can extend in different namespace. We are having similar discussion on the namespace as you suggested. I will update the namespaces as well. 3. Should i update in hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ? Since these configuration are specific to ADL file system only.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks Lei (Eddy) Xu for the comments.

          • You mentioned in the above comments. But PrivateAzureDataLakeFileSystem does not call it within synchronized calls (e.g., PrivateAzureDataLakeFileSystem#create. Although syncMap is a synchronizedMap, putFileStatus has multiple operations on syncMap, which can not guarantee atomicity.
          • It might be a better idea to provide atomicity in PrivateAzureDataLakeFileSystem. A couple of places have multiple cache calls within the same function (e.g., rename()).

          PutFileStatus has only 1 operation on syncMap. Could you please elaborate on the scenario which could be affected? To be certain, are you reviewing to HADOOP-12666-005.patch right?

          • It might be a good idea to rename FileStatusCacheManager#getFileStatus, putFileStatus, removeFileStatus to get/put/remove, because the class name already clearly indicates the context.

          Agree. Renamed to get/put/remove

          • FileStatusCacheObject can only store an absolute expiration time. And its methods can be package-level methods.

          You are right, this is an alternate approach to handle cache expiration time. I think we can leave with current implementation using time to live check, Please let me know if you find any issue with that approach?

          • I saw a few places, e.g., PrivateAzureDataLakeFileSystem#rename/delete, that clear the cache if the param is a directory. Could you justify the reason behind this? Would it cause noticeable performance degradation? Or as an alternative, using LinkedList + TreeMap for FileStatusCacheManager?

          Yes, To avoid performance & correction issue when directory is renamed/deleted. In such cases, Cache is holding stale entries and needs to be removed so that delete/rename followed by getFileStatus call (For file/folder present in the directory). At the point of folder deletion, Cache might be holding multiple FileStatus instances within directory. Its efficient to nuke the cache and rebuild it than iterate over.

          The current cache is a basic implementation to hold FileStatus instances to start with and we would continue to enhance in upcoming changes.

          • One general question, is this FileStatusCacheManager in HdfsClient? If it is the case, how do you make them consistent across clients on multiple nodes?

          FileStatusCacheManager need not be consistent across clients. FileStatusCacheManager is build based on the ListStatus and GetFileStatus calls from the respective clients.

          • Can we use Precondtions here? It will be more descriptive.

          Are you referring to com.google.common.base.Preconditions?

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks Lei (Eddy) Xu for the comments. You mentioned in the above comments. But PrivateAzureDataLakeFileSystem does not call it within synchronized calls (e.g., PrivateAzureDataLakeFileSystem#create. Although syncMap is a synchronizedMap, putFileStatus has multiple operations on syncMap, which can not guarantee atomicity. It might be a better idea to provide atomicity in PrivateAzureDataLakeFileSystem. A couple of places have multiple cache calls within the same function (e.g., rename()). PutFileStatus has only 1 operation on syncMap. Could you please elaborate on the scenario which could be affected? To be certain, are you reviewing to HADOOP-12666 -005.patch right? It might be a good idea to rename FileStatusCacheManager#getFileStatus, putFileStatus, removeFileStatus to get/put/remove, because the class name already clearly indicates the context. Agree. Renamed to get/put/remove FileStatusCacheObject can only store an absolute expiration time. And its methods can be package-level methods. You are right, this is an alternate approach to handle cache expiration time. I think we can leave with current implementation using time to live check, Please let me know if you find any issue with that approach? I saw a few places, e.g., PrivateAzureDataLakeFileSystem#rename/delete, that clear the cache if the param is a directory. Could you justify the reason behind this? Would it cause noticeable performance degradation? Or as an alternative, using LinkedList + TreeMap for FileStatusCacheManager? Yes, To avoid performance & correction issue when directory is renamed/deleted. In such cases, Cache is holding stale entries and needs to be removed so that delete/rename followed by getFileStatus call (For file/folder present in the directory). At the point of folder deletion, Cache might be holding multiple FileStatus instances within directory. Its efficient to nuke the cache and rebuild it than iterate over. The current cache is a basic implementation to hold FileStatus instances to start with and we would continue to enhance in upcoming changes. One general question, is this FileStatusCacheManager in HdfsClient? If it is the case, how do you make them consistent across clients on multiple nodes? FileStatusCacheManager need not be consistent across clients. FileStatusCacheManager is build based on the ListStatus and GetFileStatus calls from the respective clients. Can we use Precondtions here? It will be more descriptive. Are you referring to com.google.common.base.Preconditions?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thank you Chris Nauroth for the comments.

          1. Yes, i will upload the respective document.
          2. We do have extended Contact test cases which integrated with back-end service however those test are not pushed as part of this check in. I will create separate JIRA for the Live mode test cases.
          3. We do run contract test cases.

          For the code.
          1. I have refactored namespace as per comments from Aaron Fabbri and Chris Nauroth suggestion as

          Namespace Purpose
          org.apache.hadoop.fs.adl Public interface exposed for Hadoop application to integrate with. For long term support, this namespace to stay even if we remove refactor dependency on org.apache.hadoop.hdfs.web
          org.apache.hadoop.hdfs.web Extension of WebHdfsFileSystem to override protected functionality. Example ConnectionFactory access, Override redirection operation etc.

          2.

          PrivateAzureDataLakeFileSystem

          is exposed for

          AdlFileSystem

          to inherit. I will add the documentation for the same.
          3. Intentional to not add lock. Even if multiple instances are created the last instance would be used across to refresh token.
          4. Yes, Similar comment i got from Chris Douglas as well. Reason behind hiding logging through was to switch quickly between

          Log

          and

          System.out.println

          during debugging. Quickest way is change the code than configuration file. We will migrate to use SLF4J but not part of this patch release. is that fine?
          5. Explained above
          6. Agree and incorporated the code change.
          7.

          FileStatus

          cache management feature is configurable. In case of some scenarios are breaking for the customer, they can turn off the local cache. Cache scope is within the process. I will document on the behavior and tuning flags like duration of the cache. We do see great performance improvement however we do not wish to compromise on the correctness.
          8.

          ADLConfKeys#LOG_VERSION

          is to capture code instrumentation version. this information is used only during debugging session.
          9. Excellent point. Bug was, mock server hung up

          TestAAAAAUploadBenchmark/TestAAAAADownloadBenchmark

          are not executed before other tests. I will investigate the root cause of this issue since you pointed out on the execution order not guaranteed.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thank you Chris Nauroth for the comments. 1. Yes, i will upload the respective document. 2. We do have extended Contact test cases which integrated with back-end service however those test are not pushed as part of this check in. I will create separate JIRA for the Live mode test cases. 3. We do run contract test cases. For the code. 1. I have refactored namespace as per comments from Aaron Fabbri and Chris Nauroth suggestion as Namespace Purpose org.apache.hadoop.fs.adl Public interface exposed for Hadoop application to integrate with. For long term support, this namespace to stay even if we remove refactor dependency on org.apache.hadoop.hdfs.web org.apache.hadoop.hdfs.web Extension of WebHdfsFileSystem to override protected functionality. Example ConnectionFactory access, Override redirection operation etc. 2. PrivateAzureDataLakeFileSystem is exposed for AdlFileSystem to inherit. I will add the documentation for the same. 3. Intentional to not add lock. Even if multiple instances are created the last instance would be used across to refresh token. 4. Yes, Similar comment i got from Chris Douglas as well. Reason behind hiding logging through was to switch quickly between Log and System.out.println during debugging. Quickest way is change the code than configuration file. We will migrate to use SLF4J but not part of this patch release. is that fine? 5. Explained above 6. Agree and incorporated the code change. 7. FileStatus cache management feature is configurable. In case of some scenarios are breaking for the customer, they can turn off the local cache. Cache scope is within the process. I will document on the behavior and tuning flags like duration of the cache. We do see great performance improvement however we do not wish to compromise on the correctness. 8. ADLConfKeys#LOG_VERSION is to capture code instrumentation version. this information is used only during debugging session. 9. Excellent point. Bug was, mock server hung up TestAAAAAUploadBenchmark/TestAAAAADownloadBenchmark are not executed before other tests. I will investigate the root cause of this issue since you pointed out on the execution order not guaranteed.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Aaron Fabbri - Corrected alias

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Aaron Fabbri - Corrected alias
          Hide
          mackrorysd Sean Mackrory added a comment -

          One caveat to consider is overlap between configuration properties for different file systems. Generally the configuration for cloud storage uses a unique set of properties, which makes it safer / easier to have HDFS or other webhdfs services and cloud storage configured in the same cluster. This enables more diverse workloads and allows you to copy between filesystems easily. Since this implementation extends WebHDFS, there appears to be a need to overload configuration properties to use it. Should we modify this to use unique properties instead?

          Show
          mackrorysd Sean Mackrory added a comment - One caveat to consider is overlap between configuration properties for different file systems. Generally the configuration for cloud storage uses a unique set of properties, which makes it safer / easier to have HDFS or other webhdfs services and cloud storage configured in the same cluster. This enables more diverse workloads and allows you to copy between filesystems easily. Since this implementation extends WebHDFS, there appears to be a need to overload configuration properties to use it. Should we modify this to use unique properties instead?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Incorporated review comments from Chris Nauroth, Aaron Fabbri and Lei (Eddy) Xu.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Incorporated review comments from Chris Nauroth , Aaron Fabbri and Lei (Eddy) Xu .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 1m 30s trunk passed with JDK v1.8.0_72
          +1 compile 1m 23s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 1m 54s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools
          +1 findbugs 0m 0s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 40s the patch passed
          +1 compile 1m 29s the patch passed with JDK v1.8.0_72
          +1 javac 1m 29s the patch passed
          +1 compile 1m 26s the patch passed with JDK v1.7.0_95
          +1 javac 1m 26s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 2m 18s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 19s the patch passed with JDK v1.7.0_95
          +1 unit 38m 23s hadoop-tools in the patch passed with JDK v1.8.0_72.
          +1 unit 1m 36s hadoop-azure-datalake in the patch passed with JDK v1.8.0_72.
          +1 unit 37m 39s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 1m 30s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          107m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787646/HADOOP-12666-006.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux 1c2e94422e41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2fb423e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8609/testReport/
          modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8609/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 1m 30s trunk passed with JDK v1.8.0_72 +1 compile 1m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 28s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-tools +1 findbugs 0m 0s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 1m 29s the patch passed with JDK v1.8.0_72 +1 javac 1m 29s the patch passed +1 compile 1m 26s the patch passed with JDK v1.7.0_95 +1 javac 1m 26s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 2m 18s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-tools +1 findbugs 0m 37s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 19s the patch passed with JDK v1.7.0_95 +1 unit 38m 23s hadoop-tools in the patch passed with JDK v1.8.0_72. +1 unit 1m 36s hadoop-azure-datalake in the patch passed with JDK v1.8.0_72. +1 unit 37m 39s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 1m 30s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 107m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787646/HADOOP-12666-006.patch JIRA Issue HADOOP-12666 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 1c2e94422e41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2fb423e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8609/testReport/ modules C: hadoop-tools hadoop-tools/hadoop-azure-datalake U: hadoop-tools Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8609/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          Yes, I confirmed the parameters should be documented w/ defaults in that core-default.xml file. See the s3a parameters for similar example.

          Show
          fabbri Aaron Fabbri added a comment - Yes, I confirmed the parameters should be documented w/ defaults in that core-default.xml file. See the s3a parameters for similar example.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Douglas, Chris Nauroth and Aaron Fabbri - Do you have any further comments on the latest patch?

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Douglas , Chris Nauroth and Aaron Fabbri - Do you have any further comments on the latest patch?
          Hide
          fabbri Aaron Fabbri added a comment -

          Thank your for your contributions. This is a large patch with some dense spots,
          which makes it hard for folks to get time to review properly. In the future you
          should break up the work into multiple commits and associate patches with jira
          subtasks. This will make your life easier as well.

          Summary of issues, this round:

          1. Still some parts I haven't carefully reviewed due to size of patch.
          2. FileStatusCacheManager seems to have local race conditions and zero
          intra-node coherency.
          3. Seems like abuse of volatile / lack of locking in BatchByteArrayInputStream.
          4. How do Hadoop folks feel about this hadoop-tools/hadoop-azure-datalake code
          declaring classes in the hadoop.hdfs.web package? I feel it needs cleanup.
          5. Still need to put config parms in core-default.xml and make names lower case.

          There are a bunch of other comments / questions inline below. Search for "AF>"

          <snip>
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java
          new file mode 100644
          index 0000000..fd6a2ff
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java
          @@ -0,0 +1,140 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          <snip>
          + * ACID properties are maintained in overloaded api in @see
          + * PrivateAzureDataLakeFileSystem class.
          + */
          +public final class FileStatusCacheManager {
          + private static final FileStatusCacheManager FILE_STATUS_CACHE_MANAGER = new
          + FileStatusCacheManager();
          + private Map<String, FileStatusCacheObject> syncMap = null;
          +
          + /**
          + * Constructor.
          + */
          + private FileStatusCacheManager() {

          AF> This class seems to have serious issues that need addressing:

          1. Local race conditions in caller PrivateAzureDataLakeFileSystem
          2. No mechanism for cache invalidation across nodes in the cluster.

          + LinkedHashMap<String, FileStatusCacheObject> map = new
          + LinkedHashMap<String, FileStatusCacheObject>() {
          +
          + private static final int MAX_ENTRIES = 5000;
          +
          + @Override
          + protected boolean removeEldestEntry(Map.Entry eldest) {
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java
          new file mode 100644
          index 0000000..5316443
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java
          @@ -0,0 +1,59 @@
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java
          new file mode 100644
          index 0000000..a0ca4a9
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java
          @@ -0,0 +1,53 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          <snip>
          +
          +/**
          + * Create ADL filesystem delegation with Swebhdfs scheme. Intent to use by
          + * AdlFileSystem only.
          + */

          AF> Update comment? This uses "adl" scheme, right?

          +public class PrivateAzureDataLake extends DelegateToFileSystem {
          + public static final int DEFAULT_PORT = 443;

          AF> What is this class used for? I didn't see any uses.

          +
          + PrivateAzureDataLake(URI theUri, Configuration conf)
          + throws IOException, URISyntaxException {
          + super(theUri, createFileSystem(conf), conf,
          + PrivateAzureDataLakeFileSystem.SCHEME, false);
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java
          new file mode 100644
          index 0000000..db4a83c
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java
          @@ -0,0 +1,1516 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          <snip>
          + * limitations under the License.
          + *
          + */
          +
          +package org.apache.hadoop.hdfs.web;
          +
          AF> Care to comment why this is in the ..hdfs.web package instead of fs.adl?
          It lives in hadoop-tools/hadoop-azure-datalake in the source tree.

          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.fs.BlockLocation;
          <snip>
          +import java.util.concurrent.TimeUnit;
          +import java.util.concurrent.atomic.AtomicInteger;
          +
          +/**
          + * Extended @see SWebHdfsFileSystem API. This class contains Azure data lake
          + * specific stability, Reliability and performance improvement.
          + * <p>
          + * Motivation behind PrivateAzureDataLakeFileSystem

          AF> ?

          + */
          +public class PrivateAzureDataLakeFileSystem extends SWebHdfsFileSystem {
          +
          + public static final String SCHEME = "adl";
          + /**
          + * Process wide thread pool for data lake file system.
          + * Threads need to be daemon so that they dont prevent the process from
          + * exiting
          + */
          + private static final ExecutorService EXECUTOR = Executors
          + .newCachedThreadPool(new ThreadFactory() {
          + public Thread newThread(Runnable r)

          Unknown macro: { + Thread t = Executors.defaultThreadFactory().newThread(r); + t.setDaemon(true); + return t; + }

          + });
          + private static String hostName = null;
          + private static AtomicInteger metricsSourceNameCounter = new AtomicInteger();
          + // Feature configuration
          + // Publicly Exposed
          <snip>
          +
          + /**
          + * Need to override default getHomeDirectory implementation due to
          + * HDFS-8542 causing MR jobs to fail in initial

          AF> Due to the bug or due to the fix? The fix was merged in 2.8.0, right?

          + * phase. Constructing home directory locally is fine as long as hadoop
          + * local user name and ADL user name relation
          + * ship is not agreed upon.

          AF> I'm not understanding this last sentence, can you explain?

          + *
          + * @return Hadoop user home directory.
          + */
          + @Override
          + public final Path getHomeDirectory() {
          + try

          Unknown macro: { + return makeQualified(new Path( + "/user/" + UserGroupInformation.getCurrentUser().getShortUserName())); + }

          catch (IOException e)

          Unknown macro: { + }

          +
          + return new Path("/user/" + userName);
          + }
          +
          <snip>
          + */
          + @Override
          + public final boolean setReplication(final Path p, final short replication)
          + throws IOException

          Unknown macro: { + return true; + }

          +
          + /**
          + * Invoked parent setTimes default implementation only.
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance
          + *
          + * @param p File/Folder path
          + * @param mtime Modification time
          + * @param atime Access time
          + * @throws IOException when system error, internal server error or user error
          + */
          + @Override
          + public final void setTimes(final Path p, final long mtime, final long atime)
          + throws IOException {
          + if (featureCacheFileStatus)

          Unknown macro: { + String filePath = p.isAbsoluteAndSchemeAuthorityNull() ? + getUri() + p.toString() }

          + super.setTimes(p, mtime, atime);
          + }
          +
          + /**
          + * Invokes parent setPermission default implementation only.
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance
          + *
          + * @param p File/Folder path
          + * @param permission Instance FsPermission. Octal values
          + * @throws IOException when system error, internal server error or user error
          + */
          + @Override
          + public final void setPermission(final Path p, final FsPermission permission)
          + throws IOException {
          + if (featureCacheFileStatus)

          Unknown macro: { + String filePath = p.isAbsoluteAndSchemeAuthorityNull() ? + getUri() + p.toString() }

          + super.setPermission(p, permission);
          + }
          +
          + /**
          + * Avoid call to Azure data lake backend system. Look in the local cache if
          + * FileStatus from the previous call has
          + * already been cached.
          + *
          + * Cache lookup is default enable. and can be set using configuration.
          + *
          + * @param f File/Folder path
          + * @return FileStatus instance containing metadata information of f
          + * @throws IOException For any system error
          + */
          + @Override
          + public FileStatus getFileStatus(Path f) throws IOException {
          + statistics.incrementReadOps(1);
          + FileStatus status = null;
          + if (featureCacheFileStatus)

          Unknown macro: { + status = fileStatusCacheManager.get(makeQualified(f)); + }

          +
          + if (status == null)

          Unknown macro: { + status = super.getFileStatus(f); + }

          else

          Unknown macro: { + ADLLogger.log("Cached Instance Found }

          +
          + if (featureCacheFileStatus) {
          + if (fileStatusCacheManager.get(makeQualified(f)) == null)

          Unknown macro: { + fileStatusCacheManager.put(status, featureCacheFileStatusDuration); + }

          + }

          AF> Is this a race condition?

          thread 1> getFileStatus(), cache miss
          super.getStatus -> s1
          cache.get() -> null

          thread 2> delete()
          cache.clear()

          thread 1> cache.put(s1)

          Maybe provide an atomic putIfAbsent() for FileStatusCacheManager. You can
          synchronize on the underlying map object I believe (see
          Collections.synchronizedMap()).
          +
          + if (overrideOwner)

          Unknown macro: { + FileStatus proxiedStatus = new FileStatus(status.getLen(), + status.isDirectory(), status.getReplication(), status.getBlockSize(), + status.getModificationTime(), status.getAccessTime(), + status.getPermission(), userName, "hdfs", status.getPath()); + return proxiedStatus; + }

          else

          Unknown macro: { + return status; + }

          + }
          +
          + /**
          + * Invokes parent delete() default implementation only.
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance
          + *
          + * @param f File/Folder path
          + * @param recursive true if the contents within folder needs to be removed
          + * as well
          + * @return true if the delete operation is successful other false.
          + * @throws IOException For any system exception
          + */
          + @Override
          + public boolean delete(Path f, boolean recursive) throws IOException {
          + if (featureCacheFileStatus) {
          + FileStatus fs = fileStatusCacheManager.get(makeQualified(f));
          + if (fs != null && fs.isFile())

          Unknown macro: { + fileStatusCacheManager.remove(makeQualified(f)); + }

          else

          Unknown macro: { + fileStatusCacheManager.clear(); AF> Seems like there is a less-likely race condition here. (f is replaced by a directory after checking fs.isFile()) + }

          + }
          + instrumentation.fileDeleted();
          + return super.delete(f, recursive);
          + }
          +
          + /**
          + * Invokes parent rename default implementation only.
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance *
          + *
          + * @param src Source path
          + * @param dst Destination path
          + * @return True if the rename operation is successful otherwise false
          + * @throws IOException For any system error.
          + */
          + @Override
          + public boolean rename(final Path src, final Path dst) throws IOException {
          + if (featureCacheFileStatus) {
          + FileStatus fsSrc = fileStatusCacheManager.get(makeQualified(src));
          + FileStatus fsDst = fileStatusCacheManager.get(makeQualified(dst));
          +
          + if ((fsSrc != null && !fsSrc.isFile()) || (fsDst != null && !fsDst
          + .isFile()))

          Unknown macro: { + fileStatusCacheManager.clear(); + }

          else

          Unknown macro: { + fileStatusCacheManager.remove(makeQualified(src)); + fileStatusCacheManager.remove(makeQualified(dst)); + }

          + }
          + return super.rename(src, dst);

          AF> Similar pattern of get/mutate non-atomically repeats here and below.

          + }
          +
          + /**
          + * Overloaded version of rename. Invokes parent rename implementation only.
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance
          + *
          + * @param src Source path
          + * @param dst Desitnation path
          + * @param options Defined in webhdfs specification
          + * @throws IOException For system error
          + */
          + @Override
          + public void rename(Path src, Path dst, Options.Rename... options)
          + throws IOException {
          + if (featureCacheFileStatus) {
          + FileStatus fsSrc = fileStatusCacheManager.get(makeQualified(src));
          + FileStatus fsDst = fileStatusCacheManager.get(makeQualified(dst));
          +
          + if ((fsSrc != null && !fsSrc.isFile()) || (fsDst != null && !fsDst
          + .isFile()))

          else

          Unknown macro: { + fileStatusCacheManager.remove(makeQualified(src)); + fileStatusCacheManager.remove(makeQualified(dst)); + }

          + }
          + super.rename(src, dst, options);
          + }
          +
          + /**
          + * Invokes parent append default implementation
          + *
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance.
          + *
          + * @param f Stream path
          + * @return Output stream.
          + * @throws IOException For system error
          + */
          + @Override
          + public FSDataOutputStream append(Path f) throws IOException

          Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + return super.append(f); + }

          +
          + /**
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance.
          + *
          + * @param f Existing file path
          + * @param bufferSize Size of the buffer
          + * @param progress Progress indicator
          + * @return FSDataOutputStream OutputStream on which application can push
          + * stream of bytes
          + * @throws IOException For any system exception
          + */
          + @Override
          + public FSDataOutputStream append(Path f, int bufferSize,
          + Progressable progress) throws IOException

          Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + return super.append(f, bufferSize, progress); + }

          +
          + /**
          + * Removes cached FileStatus entry to maintain latest information on the
          + * FileStatus instance.
          + *
          + * @param trg Target file path
          + * @param srcs List of sources to be concatinated. ADL concatinate in the
          + * same order passed as parameter.
          + * @throws IOException For any system exception
          + */
          + @Override
          + public void concat(final Path trg, final Path[] srcs) throws IOException {
          + if (featureCacheFileStatus) {
          + String filePath = trg.isAbsoluteAndSchemeAuthorityNull() ?
          + getUri() + trg.toString() :
          + trg.toString();
          + fileStatusCacheManager.remove(new Path(filePath));
          + for (int i = 0; i < srcs.length; ++i)

          Unknown macro: { + filePath = srcs[0].isAbsoluteAndSchemeAuthorityNull() ? + getUri() + srcs[0].toString() }

          + }
          + super.concat(trg, srcs);
          + }
          +
          <snip>
          + * failures.
          + * 2. Performance boost to jobs which are slow writer, avoided network latency
          + * 3. ADL equally better performing with multiple of 4MB chunk as append
          + * calls.
          + *
          + * @param f File path
          + * @param permission Access perfrmission for the newly created file

          AF> typo

          + * @param overwrite Remove existing file and recreate new one if true
          + * otherwise throw error if file exist
          + * @param bufferSize Buffer size, ADL backend does not honour
          + * @param replication Replication count, ADL backen does not hounour
          + * @param blockSize Block size, ADL backend does not honour
          + * @param progress Progress indicator
          + * @return FSDataOutputStream OutputStream on which application can push
          + * stream of bytes
          + * @throws IOException when system error, internal server error or user error
          + */
          + @Override
          + public FSDataOutputStream create(final Path f, final FsPermission permission,
          + final boolean overwrite, final int bufferSize, final short replication,
          + final long blockSize, final Progressable progress) throws IOException {
          + statistics.incrementWriteOps(1);
          + // Increment the counter
          + instrumentation.fileCreated();
          +
          + if (featureCacheFileStatus)

          Unknown macro: { + fileStatusCacheManager.remove(makeQualified(f)); + }

          +
          + return new FSDataOutputStream(new BatchAppendOutputStream(f, bufferSize,
          + new PermissionParam(applyUMask(permission)),
          + new OverwriteParam(overwrite), new BufferSizeParam(bufferSize),
          + new ReplicationParam(replication), new BlockSizeParam(blockSize),
          + new ADLVersionInfo(getADLEnabledFeatureSet())), statistics)

          ;
          + }
          +
          + @Override
          + @SuppressWarnings("deprecation")
          + public FSDataOutputStream createNonRecursive(final Path f,
          + final FsPermission permission, final EnumSet<CreateFlag> flag,
          + final int bufferSize, final short replication, final long blockSize,
          + final Progressable progress) throws IOException {
          + statistics.incrementWriteOps(1);
          + // Increment the counter
          + instrumentation.fileCreated();
          +
          + if (featureCacheFileStatus)

          Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + }

          +
          + String leaseId = java.util.UUID.randomUUID().toString();
          + return new FSDataOutputStream(new BatchAppendOutputStream(f, bufferSize,
          + new PermissionParam(applyUMask(permission)), new CreateFlagParam(flag),
          + new CreateParentParam(false), new BufferSizeParam(bufferSize),
          + new ReplicationParam(replication), new LeaseParam(leaseId),
          + new BlockSizeParam(blockSize),
          + new ADLVersionInfo(getADLEnabledFeatureSet())), statistics)

          Unknown macro: { + }

          ;
          + }
          +
          + /**
          + * Since defined as private in parent class, redefined to pass through
          + * Create api implementation.
          + *
          + * @param permission
          + * @return FsPermission list
          + */
          + private FsPermission applyUMask(FsPermission permission) {
          + FsPermission fsPermission = permission;
          + if (fsPermission == null)

          Unknown macro: { + fsPermission = FsPermission.getDefault(); + }

          + return fsPermission.applyUMask(FsPermission.getUMask(getConf()));
          + }
          +
          + /**
          + * Open call semantic is handled differently in case of ADL. Instead of
          + * network stream is returned to the user,
          + * Overridden FsInputStream is returned.
          + *
          + * 1. No dedicated connection to server.
          + * 2. Process level concurrent read ahead Buffering is done, This allows
          + * data to be available for caller quickly.
          + * 3. Number of byte to read ahead is configurable.
          + *
          + * Advantage of Process level concurrent read ahead Buffering semantics is
          + * 1. ADL backend server does not allow idle connection for longer duration
          + * . In case of slow reader scenario,
          + * observed connection timeout/Connection reset causing occasional job
          + * failures.

          AF> Did you guys consider handling this as transparently reconnecting, instead
          of doing separate connections for each op? Seems like performance would be alot
          better?

          + * 2. Performance boost to jobs which are slow reader, avoided network latency

          AF> I'd expect you to want a connection per-thread, instead of per-op.

          + * 3. Compressed format support like ORC, and large data files gains the
          + * most out of this implementation.
          + *
          + * Read ahead feature is configurable.
          + *
          + * @param f File path
          + * @param buffersize Buffer size
          + * @return FSDataInputStream InputStream on which application can read
          + * stream of bytes
          + * @throws IOException when system error, internal server error or user error
          + */
          + @Override
          + public FSDataInputStream open(final Path f, final int buffersize)
          + throws IOException {
          + long statContructionTime = System.currentTimeMillis();
          + statistics.incrementReadOps(1);
          +
          + ADLLogger.log("statistics report Time " + (System.currentTimeMillis()
          + - statContructionTime));
          +
          + final HttpOpParam.Op op = GetOpParam.Op.OPEN;
          + // use a runner so the open can recover from an invalid token
          + FsPathConnectionRunner runner = null;
          +
          + if (featureConcurrentReadWithReadAhead)

          Unknown macro: { + long urlContructionTime = System.currentTimeMillis(); + URL url = this.toUrl(op, f, new BufferSizeParam(buffersize), + new ReadADLNoRedirectParam(true), + new ADLVersionInfo(getADLEnabledFeatureSet())); + ADLLogger.log("URL Construction Time " + (System.currentTimeMillis() + - urlContructionTime)); + + long bbContructionTime = System.currentTimeMillis(); + BatchByteArrayInputStream bb = new BatchByteArrayInputStream(url, f, + maxBufferSize, maxConcurrentConnection); + ADLLogger.log("BatchByteArrayInputStream Construction Time " + ( + System.currentTimeMillis() - bbContructionTime)); + + long finContructionTime = System.currentTimeMillis(); + FSDataInputStream fin = new FSDataInputStream(bb); + ADLLogger.log( + "FSDataInputStream Construction Time " + (System.currentTimeMillis() + - finContructionTime)); AF> This case could use some perf optimization. e.g. Three calls to get system time. + return fin; + }

          else {
          + if (featureRedirectOff)

          Unknown macro: { + long urlContructionTime = System.currentTimeMillis(); + runner = new FsPathConnectionRunner(ADLGetOpParam.Op.OPEN, f, + new BufferSizeParam(buffersize), new ReadADLNoRedirectParam(true), + new ADLVersionInfo(getADLEnabledFeatureSet())); + ADLLogger.log("Runner Construction Time " + (System.currentTimeMillis() + - urlContructionTime)); AF> How about adding ADLLogger.logWithTimestamp(). That way, if the logger is disabled, you don't keep getting system time. + }

          else

          Unknown macro: { + runner = new FsPathConnectionRunner(op, f, + new BufferSizeParam(buffersize)); + }

          +
          + return new FSDataInputStream(
          + new OffsetUrlInputStream(new UnresolvedUrlOpener(runner),
          + new OffsetUrlOpener(null)));
          + }
          + }
          +
          + /**
          + * On successful response from the server, @see FileStatusCacheManger is
          + * updated with FileStatus objects.
          + *
          + * @param f File/Folder path
          + * @return FileStatus array list
          + * @throws IOException For system error
          + */
          + @Override
          + public FileStatus[] listStatus(final Path f) throws IOException {
          + FileStatus[] fileStatuses = super.listStatus(f);
          + for (int i = 0; i < fileStatuses.length; i++) {
          + if (featureCacheFileStatus)

          Unknown macro: { + fileStatusCacheManager + .put(fileStatuses[i], featureCacheFileStatusDuration); + }

          +
          + if (overrideOwner)

          Unknown macro: { + fileStatuses[i] = new FileStatus(fileStatuses[i].getLen(), + fileStatuses[i].isDirectory(), fileStatuses[i].getReplication(), + fileStatuses[i].getBlockSize(), + fileStatuses[i].getModificationTime(), + fileStatuses[i].getAccessTime(), fileStatuses[i].getPermission(), + userName, "hdfs", fileStatuses[i].getPath()); + }

          + }
          + return fileStatuses;
          + }
          +
          + @Override
          + public BlockLocation[] getFileBlockLocations(final FileStatus status,
          + final long offset, final long length) throws IOException {
          + if (status == null)

          Unknown macro: { + return null; + }

          +
          + if (featureGetBlockLocationLocallyBundled) {
          + if ((offset < 0) || (length < 0))

          Unknown macro: { + throw new IllegalArgumentException("Invalid start or len parameter"); + }

          +
          + if (status.getLen() < offset) {
          + if (ADLLogger.isLogEnabled())

          Unknown macro: { AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations }

          + return new BlockLocation[0];
          + }
          +
          + final String[] name =

          Unknown macro: {"localhost"}

          ;
          + final String[] host =

          ;

          AF> Just use "name" twice instead of defining host?

          + long blockSize = ADLConfKeys.DEFAULT_EXTENT_SIZE;
          + if (blockSize <= 0)

          Unknown macro: { AF> Why the runtime check of a compile-time constant? How about just add a comment near the definition "must be non-zero" + throw new IllegalArgumentException( + "The block size for the given file is not a positive number}

          + int numberOfLocations =
          + (int) (length / blockSize) + ((length % blockSize == 0) ? 0 : 1);
          + BlockLocation[] locations = new BlockLocation[numberOfLocations];
          + for (int i = 0; i < locations.length; i++)

          Unknown macro: { + long currentOffset = offset + (i * blockSize); + long currentLength = Math + .min(blockSize, offset + length - currentOffset); + locations[i] = new BlockLocation(name, host, currentOffset, + currentLength); + }

          +
          + if (ADLLogger.isLogEnabled())

          Unknown macro: { AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations }

          +
          + return locations;
          +
          + } else

          Unknown macro: { + return getFileBlockLocations(status.getPath(), offset, length); + }

          + }
          +
          + @Override
          + public BlockLocation[] getFileBlockLocations(final Path p, final long offset,
          + final long length) throws IOException {
          + statistics.incrementReadOps(1);
          +
          + if (featureGetBlockLocationLocallyBundled)

          Unknown macro: { + FileStatus fileStatus = getFileStatus(p); + return getFileBlockLocations(fileStatus, offset, length); + }

          else

          Unknown macro: { + return super.getFileBlockLocations(p, offset, length); + }

          + }
          +
          + @Override
          + public synchronized void close() throws IOException

          Unknown macro: { + super.close(); + AdlFileSystemMetricsSystem.unregisterSource(metricsSourceName); + AdlFileSystemMetricsSystem.fileSystemClosed(); + }

          +
          + private String getADLEnabledFeatureSet()

          Unknown macro: { + // TODO }

          +
          + enum StreamState

          Unknown macro: { + Initial, + DataCachedInLocalBuffer, + StreamEnd + }

          +
          + class BatchAppendOutputStream extends OutputStream {
          + private Path fsPath;
          + private Param<?, ?>[] parameters;
          + private byte[] data = null;
          + private int offset = 0;
          + private long length = 0;
          + private boolean eof = false;
          + private boolean hadError = false;
          + private int bufferIndex = 0;
          + private byte[][] dataBuffers = new byte[2][];
          + private int bufSize = 0;
          + private Future<Object> flushTask = null;
          +
          + public BatchAppendOutputStream(Path path, int bufferSize,
          + Param<?, ?>... param) throws IOException {
          + if (bufferSize < (ADLConfKeys.DEFAULT_BLOCK_SIZE))

          Unknown macro: { + bufSize = ADLConfKeys.DEFAULT_BLOCK_SIZE; + }

          else

          Unknown macro: { + bufSize = bufferSize; + }

          +
          + this.fsPath = path;
          + this.parameters = param;
          + this.data = getBuffer();
          + FSDataOutputStream createStream = null;
          + try {
          + if (featureRedirectOff) {
          + CreateADLNoRedirectParam skipRedirect = new CreateADLNoRedirectParam(
          + true);
          + Param<?, ?>[] tmpParam = featureFlushWhenEOF ?
          + new Param<?, ?>[param.length + 2] :
          + new Param<?, ?>[param.length + 1];
          + System.arraycopy(param, 0, tmpParam, 0, param.length);
          + tmpParam[param.length] = skipRedirect;
          + if (featureFlushWhenEOF)

          Unknown macro: { + tmpParam[param.length + 1] = new ADLFlush(false); + }

          + createStream = new FsPathOutputStreamRunner(ADLPutOpParam.Op.CREATE,
          + fsPath, 1, tmpParam).run();
          + } else

          Unknown macro: { + createStream = new FsPathOutputStreamRunner(PutOpParam.Op.CREATE, + fsPath, 1, param).run(); + }

          + } finally {
          + if (createStream != null)

          Unknown macro: { + createStream.close(); + }

          + }
          + }
          +
          + @Override
          + public final synchronized void write(int b) throws IOException {
          + if (offset == (data.length))

          Unknown macro: { + flush(); + }

          +
          + data[offset] = (byte) b;
          + offset++;
          +
          + // Statistics will get incremented again as part of the batch updates,
          + // decrement here to avoid double value
          + if (statistics != null)

          Unknown macro: { + statistics.incrementBytesWritten(-1); + }

          + }
          +
          + @Override
          + public final synchronized void write(byte[] buf, int off, int len)
          + throws IOException {
          + int bytesToWrite = len;
          + int localOff = off;
          + int localLen = len;
          + if (localLen >= data.length) {
          + // Flush data that is already in our internal buffer
          + flush();
          +
          + // Keep committing data until we have less than our internal buffers
          + // length left
          + do {
          + try

          Unknown macro: { + commit(buf, localOff, data.length, eof); + }

          catch (IOException e)

          Unknown macro: { + hadError = true; + throw e; + }

          + localOff += data.length;
          + localLen -= data.length;
          + } while (localLen >= data.length);
          + }
          +
          + // At this point, we have less than data.length left to copy from users
          + // buffer
          + if (offset + localLen >= data.length)

          Unknown macro: { + // Users buffer has enough data left to fill our internal buffer + int bytesToCopy = data.length - offset; + System.arraycopy(buf, localOff, data, offset, bytesToCopy); + offset += bytesToCopy; + + // Flush our internal buffer asynchronously + flushAsync(); + localOff += bytesToCopy; + localLen -= bytesToCopy; + }

          +
          + if (localLen > 0)

          Unknown macro: { + // Simply copy the remainder from the users buffer into our internal + // buffer + System.arraycopy(buf, localOff, data, offset, localLen); + offset += localLen; + }

          +
          + // Statistics will get incremented again as part of the batch updates,
          + // decrement here to avoid double value
          + if (statistics != null)

          Unknown macro: { + statistics.incrementBytesWritten(-bytesToWrite); + }

          + instrumentation.rawBytesUploaded(bytesToWrite);
          + }
          +
          + @Override
          + public final synchronized void flush() throws IOException {
          + waitForOutstandingFlush();
          + if (offset > 0) {
          + try

          Unknown macro: { + commit(data, 0, offset, eof); + }

          catch (IOException e)

          Unknown macro: { + hadError = true; + throw e; + }

          + }
          +
          + offset = 0;
          + }
          +
          + @Override
          + public final synchronized void close() throws IOException {
          + //TODO : 2ns call should not cause any error and no network calls.
          + if (featureRedirectOff)

          Unknown macro: { + eof = true; + }

          +
          + boolean flushedSomething = false;
          + if (hadError)

          Unknown macro: { + // No point proceeding further since the error has occurered and + // stream would be required to upload again. + return; + }

          else

          Unknown macro: { + flushedSomething = offset > 0; + flush(); + }

          +
          + if (featureRedirectOff) {
          + // If we didn't flush anything from our internal buffer, we have to
          + // call the service again
          + // with an empty payload and flush=true in the url
          + if (!flushedSomething)

          Unknown macro: { + commit(null, 0, ADLConfKeys.KB, true); + }

          + }
          +
          + ADLLogger.log(" Total bytes Written : " + (length) + " [" + fsPath + "]");
          + }
          +
          + private void commit(byte[] buffer, int off, int len, boolean endOfFile)
          + throws IOException {
          + OutputStream out = null;
          + try {
          + if (featureRedirectOff) {
          + AppendADLNoRedirectParam skipRedirect = new AppendADLNoRedirectParam(
          + true);
          + Param<?, ?>[] tmpParam = featureFlushWhenEOF ?
          + new Param<?, ?>[parameters.length + 3] :
          + new Param<?, ?>[parameters.length + 1];
          + System.arraycopy(parameters, 0, tmpParam, 0, parameters.length);
          + tmpParam[parameters.length] = skipRedirect;
          + if (featureFlushWhenEOF)

          Unknown macro: { + tmpParam[parameters.length + 1] = new ADLFlush(endOfFile); + tmpParam[parameters.length + 2] = new OffsetParam(length); + }

          +
          + out = new FsPathOutputStreamRunner(ADLPostOpParam.Op.APPEND, fsPath,
          + len, tmpParam).run();
          + } else

          Unknown macro: { + out = new FsPathOutputStreamRunner(ADLPostOpParam.Op.APPEND, fsPath, + len, parameters).run(); + }

          +
          + if (buffer != null)

          Unknown macro: { + fileStatusCacheManager.remove(fsPath); + out.write(buffer, off, len); + length += len; + }

          + } finally {
          + if (out != null)

          Unknown macro: { + out.close(); + }

          + }
          + }
          +
          + private void flushAsync() throws IOException {
          + if (offset > 0)

          Unknown macro: { + waitForOutstandingFlush(); + + // Submit the new flush task to the executor + flushTask = EXECUTOR.submit(new CommitTask(data, offset, eof)); + + // Get a new internal buffer for the user to write + data = getBuffer(); + offset = 0; + }

          + }
          +
          + private void waitForOutstandingFlush() throws IOException {
          + if (flushTask != null) {
          + try

          Unknown macro: { + flushTask.get(); + }

          catch (InterruptedException ex)

          Unknown macro: { + throw new IOException(ex); + }

          catch (ExecutionException ex)

          Unknown macro: { + // Wrap the ExecutionException in an IOException for callers can + // only handle IOException + throw new IOException(ex); + }

          finally

          Unknown macro: { + flushTask = null; + }

          + }
          + }
          +
          + private byte[] getBuffer() {
          + // Switch between the first and second buffer
          + if (bufferIndex == 0) {
          + if (dataBuffers[0] == null)

          Unknown macro: { + dataBuffers[0] = new byte[bufSize]; + }

          +
          + bufferIndex = 1;
          + return dataBuffers[0];
          + } else {
          + if (dataBuffers[1] == null)

          Unknown macro: { + dataBuffers[1] = new byte[bufSize]; + }

          +
          + bufferIndex = 0;
          + return dataBuffers[1];
          + }
          + }
          +
          + public class CommitTask implements Callable<Object> {
          + private byte[] buff;
          + private int len;
          + private boolean eof;
          +
          + public CommitTask(byte[] buffer, int size, boolean isEnd)

          Unknown macro: { + buff = buffer; + len = size; + eof = isEnd; + }

          +
          + public final Object call() throws IOException

          Unknown macro: { + commit(buff, 0, len, eof); + return null; + }

          + }
          + }
          +
          + @SuppressWarnings("checkstyle:javadocmethod")
          + /**
          + * Read data from backend in chunks instead of persistent connection. This
          + * is to avoid slow reader causing socket
          + * timeout.
          + */ protected class BatchByteArrayInputStream extends FSInputStream {

          AF> Formatting. Missing newline above.

          +
          + private static final int SIZE4MB = 4 * 1024 * 1024;
          + private final URL runner;
          + private volatile byte[] data = null;
          + private volatile long validDataHoldingSize = 0;
          + private volatile int bufferOffset = 0;

          AF> Why volatile here? Needs comments.

          I have a feeling this is wrong and you need some synchronized blocks below
          instead.

          + private volatile long currentFileOffset = 0;
          + private volatile long nextFileOffset = 0;
          + private long fileSize = 0;
          + private String guid;
          + private StreamState state = StreamState.Initial;
          + private int maxBufferSize;
          + private int maxConcurrentConnection;
          + private Path fsPath;
          + private boolean streamIsClosed;
          + private Future[] subtasks = null;
          +
          + BatchByteArrayInputStream(URL url, Path p, int bufferSize,
          + int concurrentConnection) throws IOException {
          + this.runner = url;
          + fsPath = p;
          + FileStatus fStatus = getFileStatus(fsPath);
          + if (!fStatus.isFile())

          Unknown macro: { + throw new IOException("Cannot open the directory " + p + " for " + + "reading"); + }

          + fileSize = fStatus.getLen();
          + guid = getMachineName() + System.currentTimeMillis();

          AF> Probably unique. What is impact if this collides? Why not use a random
          java UUID?

          + this.maxBufferSize = bufferSize;
          + this.maxConcurrentConnection = concurrentConnection;
          + this.streamIsClosed = false;
          + }
          +
          + @Override
          + public final int read(long position, byte[] buffer, int offset, int length)
          + throws IOException {
          + if (streamIsClosed)

          Unknown macro: { + throw new IOException("Stream already closed"); + }

          + long oldPos = this.getPos();
          +
          + int nread1;
          + try

          Unknown macro: { + this.seek(position); + nread1 = this.read(buffer, offset, length); + }

          finally

          Unknown macro: { + this.seek(oldPos); + }

          +
          + return nread1;
          + }
          +
          + @Override
          + public final int read() throws IOException {
          + if (streamIsClosed)

          + int status = doBufferAvailabilityCheck();
          + if (status == -1)

          Unknown macro: { + return status; + }

          + int ch = data[bufferOffset++] & (0xff);

          AF> this is not thread safe, but looking at FSInputStream, it appears read() is
          supposed to be.

          Seems this class is racy in general?

          + if (statistics != null)

          Unknown macro: { + statistics.incrementBytesRead(1); + }

          + return ch;
          + }
          +
          + @Override
          + public final void readFully(long position, byte[] buffer, int offset,
          + int length) throws IOException {
          + if (streamIsClosed)

          Unknown macro: { + throw new IOException("Stream already closed"); + }

          + long startTime = System.currentTimeMillis();
          + super.readFully(position, buffer, offset, length);
          + ADLLogger.log("ReadFully1 Time Taken : " + (System.currentTimeMillis()
          + - startTime));
          + if (statistics != null)

          Unknown macro: { + statistics.incrementBytesRead(length); + }

          + instrumentation.rawBytesDownloaded(length);
          + }
          +
          + @Override
          + public final int read(byte[] b, int off, int len) throws IOException {
          + if (streamIsClosed)

          + int status = doBufferAvailabilityCheck();
          + if (status == -1)

          +
          + long startTime = System.currentTimeMillis();
          + int byteRead = 0;
          + long availableBytes = validDataHoldingSize - off;
          + long requestedBytes = bufferOffset + len - off;
          + if (requestedBytes <= availableBytes) {
          + if (b == null)

          Unknown macro: { + throw new NullPointerException(); + }

          else if (off < 0 || len < 0 || len > b.length - off)

          Unknown macro: { + throw new IndexOutOfBoundsException(); + }

          else if (len == 0)

          Unknown macro: { + return 0; + }

          +
          + ADLLogger.log("AC - [BufferOffset : " + bufferOffset + " " +
          + "CurrentFileSite : " + currentFileOffset + "] Offset : " + off
          + + " Length : " + len + " Read Time Taken : " + (
          + System.currentTimeMillis() - startTime));
          + try

          Unknown macro: { + System.arraycopy(data, bufferOffset, b, off, len); + }

          catch (ArrayIndexOutOfBoundsException e)

          Unknown macro: { + ADLLogger.log("ArrayIndexOutOfBoundsException AC - [BufferOffset }

          + bufferOffset += len;
          + byteRead = len;
          + } else

          Unknown macro: { + ADLLogger.log("HC - [BufferOffset }

          + if (statistics != null)

          Unknown macro: { + statistics.incrementBytesRead(byteRead); + }

          + instrumentation.rawBytesDownloaded(byteRead);
          + return byteRead;
          + }
          +
          + private int doBufferAvailabilityCheck() throws IOException {
          + if (state == StreamState.Initial)

          Unknown macro: { + ADLLogger.log("Initial Fill"); + validDataHoldingSize = fill(nextFileOffset); + }

          +
          + long dataReloadSize = 0;
          + switch ((int) validDataHoldingSize) {
          + case -1:
          + state = StreamState.StreamEnd;
          + return -1;
          + case 0:
          + dataReloadSize = fill(nextFileOffset);
          + if (dataReloadSize <= 0)

          Unknown macro: { + state = StreamState.StreamEnd; + return (int) dataReloadSize; + }

          else

          Unknown macro: { + validDataHoldingSize = dataReloadSize; + }

          + break;
          + default:
          + break;
          + }
          +
          + if (bufferOffset >= validDataHoldingSize)

          Unknown macro: { + dataReloadSize = fill(nextFileOffset); + }

          +
          + if (bufferOffset >= ((dataReloadSize == 0) ?
          + validDataHoldingSize :
          + dataReloadSize))

          Unknown macro: { + state = StreamState.StreamEnd; + return -1; + }

          +
          + validDataHoldingSize = ((dataReloadSize == 0) ?
          + validDataHoldingSize :
          + dataReloadSize);
          + state = StreamState.DataCachedInLocalBuffer;
          + return 0;

          AF> I didn't have time to review this part thoroughly.. Can take
          better look next round.

          + }
          +
          + public final long fill(final long off) throws IOException {
          + ADLLogger.log("Fill from " + off);
          + long startTime = System.currentTimeMillis();
          + if (state == StreamState.StreamEnd)

          Unknown macro: { + return -1; + }

          +
          + if (fileSize <= off)

          + int len = maxBufferSize;
          + long fileOffset = 0;
          + boolean isEntireFileCached = true;
          + if ((fileSize < maxBufferSize))

          Unknown macro: { + len = (int) fileSize; + currentFileOffset = 0; + nextFileOffset = 0; + }

          else {
          + if (len > (fileSize - off))

          Unknown macro: { + len = (int) (fileSize - off); + }

          +
          + if (BufferManager.getInstance()
          + .hasValidDataForOffset(fsPath.toString(), off))

          Unknown macro: { + len = (int) ( + BufferManager.getInstance().getBufferOffset() + BufferManager + .getInstance().getBufferSize() - (int) off); + }

          +
          + if (len <= 0)

          Unknown macro: { + len = maxBufferSize; + }

          + fileOffset = off;
          + isEntireFileCached = false;
          + }
          +
          + data = null;
          + BufferManager bm = BufferManager.getInstance();
          + data = bm.getEmpty(len);
          + boolean fetchDataOverNetwork = false;
          + if (bm.hasData(fsPath.toString(), fileOffset, len)) {
          + try

          Unknown macro: { + bm.get(data, fileOffset); + validDataHoldingSize = data.length; + currentFileOffset = fileOffset; + }

          catch (ArrayIndexOutOfBoundsException e)

          Unknown macro: { + fetchDataOverNetwork = true; + }

          + } else

          Unknown macro: { + fetchDataOverNetwork = true; + }

          +
          + if (fetchDataOverNetwork) {
          + int splitSize = getSplitSize(len);
          + try

          Unknown macro: { + validDataHoldingSize = fillDataConcurrently(data, len, fileOffset, + splitSize); + }

          catch (InterruptedException e)

          Unknown macro: { + throw new IOException(e.getMessage()); + }

          + bm.add(data, fsPath.toString(), fileOffset);
          + currentFileOffset = nextFileOffset;
          + }
          +
          + nextFileOffset += validDataHoldingSize;
          + state = StreamState.DataCachedInLocalBuffer;
          + bufferOffset = isEntireFileCached ? (int) off : 0;
          + ADLLogger.log("Buffer Refill Time Taken : " + (System.currentTimeMillis()
          + - startTime));
          + return validDataHoldingSize;
          + }
          +
          + int getSplitSize(int size) {
          + if (size <= SIZE4MB)

          Unknown macro: { + return 1; + }

          +
          + // Not practical
          + if (size > maxBufferSize)

          Unknown macro: { + size = maxBufferSize; + }

          +
          + int equalBufferSplit = size / SIZE4MB;
          + int splitSize = Math.min(equalBufferSplit, maxConcurrentConnection);
          + return splitSize;
          + }
          +
          + @Override
          + public final void seek(long pos) throws IOException {
          + if (pos == -1)

          Unknown macro: { + throw new IOException("Bad offset, cannot seek to " + pos); + }

          +
          + BufferManager bm = BufferManager.getInstance();
          + if (bm.hasValidDataForOffset(fsPath.toString(), pos))

          Unknown macro: { + state = StreamState.DataCachedInLocalBuffer; + }

          else if (pos >= 0)

          Unknown macro: { + state = StreamState.Initial; + }

          +
          + long availableBytes = (currentFileOffset + validDataHoldingSize);
          + ADLLogger.log("SEEK : " + pos + " Available " + currentFileOffset + " " +
          + "Count " + availableBytes);
          +
          + // Check if this position falls under buffered data
          + if (pos < currentFileOffset || availableBytes <= 0)

          Unknown macro: { + validDataHoldingSize = 0; + currentFileOffset = pos; + nextFileOffset = pos; + bufferOffset = 0; + return; + }

          +
          + if (pos < availableBytes && pos >= currentFileOffset)

          Unknown macro: { + state = StreamState.DataCachedInLocalBuffer; + bufferOffset = (int) (pos - currentFileOffset); + }

          else

          Unknown macro: { + validDataHoldingSize = 0; + currentFileOffset = pos; + nextFileOffset = pos; + bufferOffset = 0; + }

          + }
          +
          + @Override
          + public final long getPos() throws IOException {
          + if (streamIsClosed)

          Unknown macro: { + throw new IOException("Stream already closed"); + }

          + return bufferOffset + currentFileOffset;
          + }
          +
          + @Override
          + public final int available() throws IOException {
          + if (streamIsClosed)

          + return Integer.MAX_VALUE;
          + }
          +
          + @Override
          + public final boolean seekToNewSource(long targetPos) throws IOException

          Unknown macro: { + return false; + }

          +
          + @SuppressWarnings("unchecked")
          + public final int fillDataConcurrently(byte[] byteArray, int length,
          + long globalOffset, int splitSize)
          + throws IOException, InterruptedException {
          + ADLLogger.log("Fill up Data from " + globalOffset + " len : " + length);
          + ExecutorService executor = Executors.newFixedThreadPool(splitSize);
          + subtasks = new Future[splitSize];
          + for (int i = 0; i < splitSize; i++)

          Unknown macro: { + int offset = i * (length / splitSize); + int splitLength = (splitSize == (i + 1)) ? + (length / splitSize) + (length % splitSize) }

          +
          + executor.shutdown();

          AF> Is this shutdown() needed?

          + // wait until all tasks are finished
          + try

          Unknown macro: { + executor.awaitTermination(ADLConfKeys.DEFAULT_TIMEOUT_IN_SECONDS, + TimeUnit.SECONDS); AF> Curious, why don't you care about timeout expiring? Can't you just remove this whole try-catch block? The futures are what are ensuring tasks finish below, no? + }

          catch (InterruptedException e)

          Unknown macro: { + ADLLogger.log("Interupted }

          +
          + int totalBytePainted = 0;
          + for (int i = 0; i < splitSize; ++i) {
          + try

          Unknown macro: { + totalBytePainted += (Integer) subtasks[i].get(); + }

          catch (InterruptedException e)

          Unknown macro: { + throw new IOException(e.getCause()); + }

          catch (ExecutionException e)

          + }
          +
          + if (totalBytePainted != length)

          Unknown macro: { + throw new IOException("Expected " + length + " bytes, Got " + + totalBytePainted + " bytes"); + }

          +
          + return totalBytePainted;
          + }
          +
          + @Override
          + public final void close() throws IOException

          Unknown macro: { + BufferManager.getInstance().clear(); + + //need to cleanup the above code the stream and connection close doesnt + // happen here + //flag set to mark close happened, cannot use the stream once closed + streamIsClosed = true; + }

          +

          AF> Add function comment w/ params and @return explanation?

          + private int fillUpData(byte[] buffer, int offset, int length,
          + long globalOffset) throws IOException {
          + int totalBytesRead = 0;
          + final URL offsetUrl = new URL(
          + runner + "&" + new OffsetParam(String.valueOf(globalOffset)) + "&"
          + + new LengthParam(String.valueOf(length)) + "&openid=" + guid);

          AF> Can you explain what this openid does?

          + HttpURLConnection conn = new URLRunner(GetOpParam.Op.OPEN, offsetUrl,
          + true).run();
          + InputStream in = conn.getInputStream();
          + try {
          + int bytesRead = 0;
          + while ((bytesRead = in.read(buffer, (int) offset + totalBytesRead,
          + (int) (length - totalBytesRead))) > 0)

          Unknown macro: { + totalBytesRead += bytesRead; + }

          +
          + // InputStream must be fully consumed to enable http keep-alive
          + if (bytesRead == 0) {
          + // Looking for EOF marker byte needs to be read.
          + if (in.read() != -1)

          Unknown macro: { + throw new SocketException( + "Server returned more than requested " + "data."); AF> humm.. we expect in.read() to return 0, and then return -1? Also, you can remove the string concatenation in the exception message. + }

          + }
          + } finally

          Unknown macro: { + in.close(); + conn.disconnect(); + }

          +
          + return totalBytesRead;
          + }
          +
          + private class BackgroundReadThread implements Callable {
          +
          + private final byte[] data;
          + private int offset;
          + private int length;
          + private long globalOffset;
          +
          + BackgroundReadThread(byte[] buffer, int off, int size, long position)

          Unknown macro: { + this.data = buffer; + this.offset = off; + this.length = size; + this.globalOffset = position; + }

          +
          + public Object call() throws IOException

          Unknown macro: { + return fillUpData(data, offset, length, globalOffset); + }

          + }
          + }
          +}
          \ No newline at end of file
          diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java
          new file mode 100644
          index 0000000..a3af0f3
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java
          @@ -0,0 +1,53 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          <snip>
          + * limitations under the License.
          + *
          + */
          +
          +package org.apache.hadoop.hdfs.web;

          AF> As noted before, using filesystem types to acheive logging tags is awkward.
          Also question why this is not in org.apache.hadoop.fs.adl

          +
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.fs.DelegateToFileSystem;
          +
          +import java.io.IOException;
          +import java.net.URI;
          +import java.net.URISyntaxException;
          +
          +/**
          + * Use this class implementation to log debug information as part of the
          + * debug logs. Used during development.
          + */
          +public class PrivateDebugAzureDataLake extends DelegateToFileSystem {
          <snip>
          <snip>
          iff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java
          new file mode 100644
          index 0000000..c3e14f6
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java
          @@ -0,0 +1,63 @@
          +/*
          <snip>
          +/**
          + * Query parameter to notify backend server that the all the data has been
          + * pushed to over the stream.
          + *
          + * Used in operation code Create and Append.
          + */
          +public class ADLFlush extends BooleanParam {
          + /**

          AF> I did not review all the HTTP Op Param stuff... someone should.
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
          new file mode 100644
          index 0000000..4dbc27f
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
          @@ -0,0 +1,246 @@
          <snip>
          +## <a name="Introduction" />Introduction
          +
          +The hadoop-azure-datalake module provides support for integration with
          +[Azure Data Lake Store](https://azure.microsoft.com/en-in/services/data-lake-store/).
          +The jar file is named azure-datalake-store.jar.
          +

          AF> Thank you for the documentation.

          <snip>
          +#### <a name="#FileStatus_Cache_Management" />FileStatus Cache Management
          +Performance is one of the key features of Azure Data Lake Storage service. In order to gain a performance boost, hadoop-azure-datalake module provides basic FileStatus cache management on the client. This reduces the number of REST calls to the backend service.
          +
          +FileStatus cache scope is per process and shared between multiple `AdlFileSystem` instances within the process. FileStatus cache is built using ListStatus and GetFileStatus calls to Azure Data Lake Storage. The life of each FileStatus cached object is limited and default is 5 seconds. Time to live FileStatus cached object is configurable through core-site.xml.
          +
          +*This is an expermental feature and should be turned off for an unexpected behaviour observed during ListStatus and GetFileStatus operation.*
          +

          AF> Seems more than experimental.

          +To Enable/Disable FileStatus cache management
          +
          + <property>
          + <name>adl.feature.override.cache.filestatus</name>
          + <value>true</value>
          <snip>
          + <property>
          + <name>dfs.webhdfs.oauth2.refresh.token</name>
          + <value></value>
          + </property>
          +
          +For ADL FileSystem to take effect. Update core-site.xml with
          +
          + <property>
          + <name>fs.adl.impl</name>
          + <value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
          + </property>
          +
          + <property>
          + <name>fs.AbstractFileSystem.adl.impl</name>
          + <value>org.apache.hadoop.fs.adl.Adl</value>
          + </property>

          AF> Again, need to put these in core-default.xml with documentation and
          defaults, and always use lowercase in the property names.

          +
          +
          +### <a name="Accessing_adl_URLs" />Accessing adl URLs
          +
          +After credentials are configured in core-site.xml, any Hadoop component may
          <snip>
          diff --git hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java
          new file mode 100644
          index 0000000..1aec96c
          — /dev/null
          +++ hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java

          AF> Thank you for including tests. I did not have time to review the test code
          this round.

          Show
          fabbri Aaron Fabbri added a comment - Thank your for your contributions. This is a large patch with some dense spots, which makes it hard for folks to get time to review properly. In the future you should break up the work into multiple commits and associate patches with jira subtasks. This will make your life easier as well. Summary of issues, this round: 1. Still some parts I haven't carefully reviewed due to size of patch. 2. FileStatusCacheManager seems to have local race conditions and zero intra-node coherency. 3. Seems like abuse of volatile / lack of locking in BatchByteArrayInputStream. 4. How do Hadoop folks feel about this hadoop-tools/hadoop-azure-datalake code declaring classes in the hadoop.hdfs.web package? I feel it needs cleanup. 5. Still need to put config parms in core-default.xml and make names lower case. There are a bunch of other comments / questions inline below. Search for "AF>" <snip> <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java new file mode 100644 index 0000000..fd6a2ff — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheManager.java @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one <snip> + * ACID properties are maintained in overloaded api in @see + * PrivateAzureDataLakeFileSystem class. + */ +public final class FileStatusCacheManager { + private static final FileStatusCacheManager FILE_STATUS_CACHE_MANAGER = new + FileStatusCacheManager(); + private Map<String, FileStatusCacheObject> syncMap = null; + + /** + * Constructor. + */ + private FileStatusCacheManager() { AF> This class seems to have serious issues that need addressing: 1. Local race conditions in caller PrivateAzureDataLakeFileSystem 2. No mechanism for cache invalidation across nodes in the cluster. + LinkedHashMap<String, FileStatusCacheObject> map = new + LinkedHashMap<String, FileStatusCacheObject>() { + + private static final int MAX_ENTRIES = 5000; + + @Override + protected boolean removeEldestEntry(Map.Entry eldest) { <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java new file mode 100644 index 0000000..5316443 — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/FileStatusCacheObject.java @@ -0,0 +1,59 @@ <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java new file mode 100644 index 0000000..a0ca4a9 — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLake.java @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one <snip> + +/** + * Create ADL filesystem delegation with Swebhdfs scheme. Intent to use by + * AdlFileSystem only. + */ AF> Update comment? This uses "adl" scheme, right? +public class PrivateAzureDataLake extends DelegateToFileSystem { + public static final int DEFAULT_PORT = 443; AF> What is this class used for? I didn't see any uses. + + PrivateAzureDataLake(URI theUri, Configuration conf) + throws IOException, URISyntaxException { + super(theUri, createFileSystem(conf), conf, + PrivateAzureDataLakeFileSystem.SCHEME, false); <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java new file mode 100644 index 0000000..db4a83c — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java @@ -0,0 +1,1516 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one <snip> + * limitations under the License. + * + */ + +package org.apache.hadoop.hdfs.web; + AF> Care to comment why this is in the ..hdfs.web package instead of fs.adl? It lives in hadoop-tools/hadoop-azure-datalake in the source tree. +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.BlockLocation; <snip> +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Extended @see SWebHdfsFileSystem API. This class contains Azure data lake + * specific stability, Reliability and performance improvement. + * <p> + * Motivation behind PrivateAzureDataLakeFileSystem AF> ? + */ +public class PrivateAzureDataLakeFileSystem extends SWebHdfsFileSystem { + + public static final String SCHEME = "adl"; + /** + * Process wide thread pool for data lake file system. + * Threads need to be daemon so that they dont prevent the process from + * exiting + */ + private static final ExecutorService EXECUTOR = Executors + .newCachedThreadPool(new ThreadFactory() { + public Thread newThread(Runnable r) Unknown macro: { + Thread t = Executors.defaultThreadFactory().newThread(r); + t.setDaemon(true); + return t; + } + }); + private static String hostName = null; + private static AtomicInteger metricsSourceNameCounter = new AtomicInteger(); + // Feature configuration + // Publicly Exposed <snip> + + /** + * Need to override default getHomeDirectory implementation due to + * HDFS-8542 causing MR jobs to fail in initial AF> Due to the bug or due to the fix? The fix was merged in 2.8.0, right? + * phase. Constructing home directory locally is fine as long as hadoop + * local user name and ADL user name relation + * ship is not agreed upon. AF> I'm not understanding this last sentence, can you explain? + * + * @return Hadoop user home directory. + */ + @Override + public final Path getHomeDirectory() { + try Unknown macro: { + return makeQualified(new Path( + "/user/" + UserGroupInformation.getCurrentUser().getShortUserName())); + } catch (IOException e) Unknown macro: { + } + + return new Path("/user/" + userName); + } + <snip> + */ + @Override + public final boolean setReplication(final Path p, final short replication) + throws IOException Unknown macro: { + return true; + } + + /** + * Invoked parent setTimes default implementation only. + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance + * + * @param p File/Folder path + * @param mtime Modification time + * @param atime Access time + * @throws IOException when system error, internal server error or user error + */ + @Override + public final void setTimes(final Path p, final long mtime, final long atime) + throws IOException { + if (featureCacheFileStatus) Unknown macro: { + String filePath = p.isAbsoluteAndSchemeAuthorityNull() ? + getUri() + p.toString() } + super.setTimes(p, mtime, atime); + } + + /** + * Invokes parent setPermission default implementation only. + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance + * + * @param p File/Folder path + * @param permission Instance FsPermission. Octal values + * @throws IOException when system error, internal server error or user error + */ + @Override + public final void setPermission(final Path p, final FsPermission permission) + throws IOException { + if (featureCacheFileStatus) Unknown macro: { + String filePath = p.isAbsoluteAndSchemeAuthorityNull() ? + getUri() + p.toString() } + super.setPermission(p, permission); + } + + /** + * Avoid call to Azure data lake backend system. Look in the local cache if + * FileStatus from the previous call has + * already been cached. + * + * Cache lookup is default enable. and can be set using configuration. + * + * @param f File/Folder path + * @return FileStatus instance containing metadata information of f + * @throws IOException For any system error + */ + @Override + public FileStatus getFileStatus(Path f) throws IOException { + statistics.incrementReadOps(1); + FileStatus status = null; + if (featureCacheFileStatus) Unknown macro: { + status = fileStatusCacheManager.get(makeQualified(f)); + } + + if (status == null) Unknown macro: { + status = super.getFileStatus(f); + } else Unknown macro: { + ADLLogger.log("Cached Instance Found } + + if (featureCacheFileStatus) { + if (fileStatusCacheManager.get(makeQualified(f)) == null) Unknown macro: { + fileStatusCacheManager.put(status, featureCacheFileStatusDuration); + } + } AF> Is this a race condition? thread 1> getFileStatus(), cache miss super.getStatus -> s1 cache.get() -> null thread 2> delete() cache.clear() thread 1> cache.put(s1) Maybe provide an atomic putIfAbsent() for FileStatusCacheManager. You can synchronize on the underlying map object I believe (see Collections.synchronizedMap()). + + if (overrideOwner) Unknown macro: { + FileStatus proxiedStatus = new FileStatus(status.getLen(), + status.isDirectory(), status.getReplication(), status.getBlockSize(), + status.getModificationTime(), status.getAccessTime(), + status.getPermission(), userName, "hdfs", status.getPath()); + return proxiedStatus; + } else Unknown macro: { + return status; + } + } + + /** + * Invokes parent delete() default implementation only. + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance + * + * @param f File/Folder path + * @param recursive true if the contents within folder needs to be removed + * as well + * @return true if the delete operation is successful other false. + * @throws IOException For any system exception + */ + @Override + public boolean delete(Path f, boolean recursive) throws IOException { + if (featureCacheFileStatus) { + FileStatus fs = fileStatusCacheManager.get(makeQualified(f)); + if (fs != null && fs.isFile()) Unknown macro: { + fileStatusCacheManager.remove(makeQualified(f)); + } else Unknown macro: { + fileStatusCacheManager.clear(); AF> Seems like there is a less-likely race condition here. (f is replaced by a directory after checking fs.isFile()) + } + } + instrumentation.fileDeleted(); + return super.delete(f, recursive); + } + + /** + * Invokes parent rename default implementation only. + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance * + * + * @param src Source path + * @param dst Destination path + * @return True if the rename operation is successful otherwise false + * @throws IOException For any system error. + */ + @Override + public boolean rename(final Path src, final Path dst) throws IOException { + if (featureCacheFileStatus) { + FileStatus fsSrc = fileStatusCacheManager.get(makeQualified(src)); + FileStatus fsDst = fileStatusCacheManager.get(makeQualified(dst)); + + if ((fsSrc != null && !fsSrc.isFile()) || (fsDst != null && !fsDst + .isFile())) Unknown macro: { + fileStatusCacheManager.clear(); + } else Unknown macro: { + fileStatusCacheManager.remove(makeQualified(src)); + fileStatusCacheManager.remove(makeQualified(dst)); + } + } + return super.rename(src, dst); AF> Similar pattern of get/mutate non-atomically repeats here and below. + } + + /** + * Overloaded version of rename. Invokes parent rename implementation only. + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance + * + * @param src Source path + * @param dst Desitnation path + * @param options Defined in webhdfs specification + * @throws IOException For system error + */ + @Override + public void rename(Path src, Path dst, Options.Rename... options) + throws IOException { + if (featureCacheFileStatus) { + FileStatus fsSrc = fileStatusCacheManager.get(makeQualified(src)); + FileStatus fsDst = fileStatusCacheManager.get(makeQualified(dst)); + + if ((fsSrc != null && !fsSrc.isFile()) || (fsDst != null && !fsDst + .isFile())) else Unknown macro: { + fileStatusCacheManager.remove(makeQualified(src)); + fileStatusCacheManager.remove(makeQualified(dst)); + } + } + super.rename(src, dst, options); + } + + /** + * Invokes parent append default implementation + * + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance. + * + * @param f Stream path + * @return Output stream. + * @throws IOException For system error + */ + @Override + public FSDataOutputStream append(Path f) throws IOException Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + return super.append(f); + } + + /** + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance. + * + * @param f Existing file path + * @param bufferSize Size of the buffer + * @param progress Progress indicator + * @return FSDataOutputStream OutputStream on which application can push + * stream of bytes + * @throws IOException For any system exception + */ + @Override + public FSDataOutputStream append(Path f, int bufferSize, + Progressable progress) throws IOException Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + return super.append(f, bufferSize, progress); + } + + /** + * Removes cached FileStatus entry to maintain latest information on the + * FileStatus instance. + * + * @param trg Target file path + * @param srcs List of sources to be concatinated. ADL concatinate in the + * same order passed as parameter. + * @throws IOException For any system exception + */ + @Override + public void concat(final Path trg, final Path[] srcs) throws IOException { + if (featureCacheFileStatus) { + String filePath = trg.isAbsoluteAndSchemeAuthorityNull() ? + getUri() + trg.toString() : + trg.toString(); + fileStatusCacheManager.remove(new Path(filePath)); + for (int i = 0; i < srcs.length; ++i) Unknown macro: { + filePath = srcs[0].isAbsoluteAndSchemeAuthorityNull() ? + getUri() + srcs[0].toString() } + } + super.concat(trg, srcs); + } + <snip> + * failures. + * 2. Performance boost to jobs which are slow writer, avoided network latency + * 3. ADL equally better performing with multiple of 4MB chunk as append + * calls. + * + * @param f File path + * @param permission Access perfrmission for the newly created file AF> typo + * @param overwrite Remove existing file and recreate new one if true + * otherwise throw error if file exist + * @param bufferSize Buffer size, ADL backend does not honour + * @param replication Replication count, ADL backen does not hounour + * @param blockSize Block size, ADL backend does not honour + * @param progress Progress indicator + * @return FSDataOutputStream OutputStream on which application can push + * stream of bytes + * @throws IOException when system error, internal server error or user error + */ + @Override + public FSDataOutputStream create(final Path f, final FsPermission permission, + final boolean overwrite, final int bufferSize, final short replication, + final long blockSize, final Progressable progress) throws IOException { + statistics.incrementWriteOps(1); + // Increment the counter + instrumentation.fileCreated(); + + if (featureCacheFileStatus) Unknown macro: { + fileStatusCacheManager.remove(makeQualified(f)); + } + + return new FSDataOutputStream(new BatchAppendOutputStream(f, bufferSize, + new PermissionParam(applyUMask(permission)), + new OverwriteParam(overwrite), new BufferSizeParam(bufferSize), + new ReplicationParam(replication), new BlockSizeParam(blockSize), + new ADLVersionInfo(getADLEnabledFeatureSet())), statistics) ; + } + + @Override + @SuppressWarnings("deprecation") + public FSDataOutputStream createNonRecursive(final Path f, + final FsPermission permission, final EnumSet<CreateFlag> flag, + final int bufferSize, final short replication, final long blockSize, + final Progressable progress) throws IOException { + statistics.incrementWriteOps(1); + // Increment the counter + instrumentation.fileCreated(); + + if (featureCacheFileStatus) Unknown macro: { + String filePath = makeQualified(f).toString(); + fileStatusCacheManager.remove(new Path(filePath)); + } + + String leaseId = java.util.UUID.randomUUID().toString(); + return new FSDataOutputStream(new BatchAppendOutputStream(f, bufferSize, + new PermissionParam(applyUMask(permission)), new CreateFlagParam(flag), + new CreateParentParam(false), new BufferSizeParam(bufferSize), + new ReplicationParam(replication), new LeaseParam(leaseId), + new BlockSizeParam(blockSize), + new ADLVersionInfo(getADLEnabledFeatureSet())), statistics) Unknown macro: { + } ; + } + + /** + * Since defined as private in parent class, redefined to pass through + * Create api implementation. + * + * @param permission + * @return FsPermission list + */ + private FsPermission applyUMask(FsPermission permission) { + FsPermission fsPermission = permission; + if (fsPermission == null) Unknown macro: { + fsPermission = FsPermission.getDefault(); + } + return fsPermission.applyUMask(FsPermission.getUMask(getConf())); + } + + /** + * Open call semantic is handled differently in case of ADL. Instead of + * network stream is returned to the user, + * Overridden FsInputStream is returned. + * + * 1. No dedicated connection to server. + * 2. Process level concurrent read ahead Buffering is done, This allows + * data to be available for caller quickly. + * 3. Number of byte to read ahead is configurable. + * + * Advantage of Process level concurrent read ahead Buffering semantics is + * 1. ADL backend server does not allow idle connection for longer duration + * . In case of slow reader scenario, + * observed connection timeout/Connection reset causing occasional job + * failures. AF> Did you guys consider handling this as transparently reconnecting, instead of doing separate connections for each op? Seems like performance would be alot better? + * 2. Performance boost to jobs which are slow reader, avoided network latency AF> I'd expect you to want a connection per-thread, instead of per-op. + * 3. Compressed format support like ORC, and large data files gains the + * most out of this implementation. + * + * Read ahead feature is configurable. + * + * @param f File path + * @param buffersize Buffer size + * @return FSDataInputStream InputStream on which application can read + * stream of bytes + * @throws IOException when system error, internal server error or user error + */ + @Override + public FSDataInputStream open(final Path f, final int buffersize) + throws IOException { + long statContructionTime = System.currentTimeMillis(); + statistics.incrementReadOps(1); + + ADLLogger.log("statistics report Time " + (System.currentTimeMillis() + - statContructionTime)); + + final HttpOpParam.Op op = GetOpParam.Op.OPEN; + // use a runner so the open can recover from an invalid token + FsPathConnectionRunner runner = null; + + if (featureConcurrentReadWithReadAhead) Unknown macro: { + long urlContructionTime = System.currentTimeMillis(); + URL url = this.toUrl(op, f, new BufferSizeParam(buffersize), + new ReadADLNoRedirectParam(true), + new ADLVersionInfo(getADLEnabledFeatureSet())); + ADLLogger.log("URL Construction Time " + (System.currentTimeMillis() + - urlContructionTime)); + + long bbContructionTime = System.currentTimeMillis(); + BatchByteArrayInputStream bb = new BatchByteArrayInputStream(url, f, + maxBufferSize, maxConcurrentConnection); + ADLLogger.log("BatchByteArrayInputStream Construction Time " + ( + System.currentTimeMillis() - bbContructionTime)); + + long finContructionTime = System.currentTimeMillis(); + FSDataInputStream fin = new FSDataInputStream(bb); + ADLLogger.log( + "FSDataInputStream Construction Time " + (System.currentTimeMillis() + - finContructionTime)); AF> This case could use some perf optimization. e.g. Three calls to get system time. + return fin; + } else { + if (featureRedirectOff) Unknown macro: { + long urlContructionTime = System.currentTimeMillis(); + runner = new FsPathConnectionRunner(ADLGetOpParam.Op.OPEN, f, + new BufferSizeParam(buffersize), new ReadADLNoRedirectParam(true), + new ADLVersionInfo(getADLEnabledFeatureSet())); + ADLLogger.log("Runner Construction Time " + (System.currentTimeMillis() + - urlContructionTime)); AF> How about adding ADLLogger.logWithTimestamp(). That way, if the logger is disabled, you don't keep getting system time. + } else Unknown macro: { + runner = new FsPathConnectionRunner(op, f, + new BufferSizeParam(buffersize)); + } + + return new FSDataInputStream( + new OffsetUrlInputStream(new UnresolvedUrlOpener(runner), + new OffsetUrlOpener(null))); + } + } + + /** + * On successful response from the server, @see FileStatusCacheManger is + * updated with FileStatus objects. + * + * @param f File/Folder path + * @return FileStatus array list + * @throws IOException For system error + */ + @Override + public FileStatus[] listStatus(final Path f) throws IOException { + FileStatus[] fileStatuses = super.listStatus(f); + for (int i = 0; i < fileStatuses.length; i++) { + if (featureCacheFileStatus) Unknown macro: { + fileStatusCacheManager + .put(fileStatuses[i], featureCacheFileStatusDuration); + } + + if (overrideOwner) Unknown macro: { + fileStatuses[i] = new FileStatus(fileStatuses[i].getLen(), + fileStatuses[i].isDirectory(), fileStatuses[i].getReplication(), + fileStatuses[i].getBlockSize(), + fileStatuses[i].getModificationTime(), + fileStatuses[i].getAccessTime(), fileStatuses[i].getPermission(), + userName, "hdfs", fileStatuses[i].getPath()); + } + } + return fileStatuses; + } + + @Override + public BlockLocation[] getFileBlockLocations(final FileStatus status, + final long offset, final long length) throws IOException { + if (status == null) Unknown macro: { + return null; + } + + if (featureGetBlockLocationLocallyBundled) { + if ((offset < 0) || (length < 0)) Unknown macro: { + throw new IllegalArgumentException("Invalid start or len parameter"); + } + + if (status.getLen() < offset) { + if (ADLLogger.isLogEnabled()) Unknown macro: { AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations } + return new BlockLocation [0] ; + } + + final String[] name = Unknown macro: {"localhost"} ; + final String[] host = ; AF> Just use "name" twice instead of defining host? + long blockSize = ADLConfKeys.DEFAULT_EXTENT_SIZE; + if (blockSize <= 0) Unknown macro: { AF> Why the runtime check of a compile-time constant? How about just add a comment near the definition "must be non-zero" + throw new IllegalArgumentException( + "The block size for the given file is not a positive number} + int numberOfLocations = + (int) (length / blockSize) + ((length % blockSize == 0) ? 0 : 1); + BlockLocation[] locations = new BlockLocation [numberOfLocations] ; + for (int i = 0; i < locations.length; i++) Unknown macro: { + long currentOffset = offset + (i * blockSize); + long currentLength = Math + .min(blockSize, offset + length - currentOffset); + locations[i] = new BlockLocation(name, host, currentOffset, + currentLength); + } + + if (ADLLogger.isLogEnabled()) Unknown macro: { AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations } + + return locations; + + } else Unknown macro: { + return getFileBlockLocations(status.getPath(), offset, length); + } + } + + @Override + public BlockLocation[] getFileBlockLocations(final Path p, final long offset, + final long length) throws IOException { + statistics.incrementReadOps(1); + + if (featureGetBlockLocationLocallyBundled) Unknown macro: { + FileStatus fileStatus = getFileStatus(p); + return getFileBlockLocations(fileStatus, offset, length); + } else Unknown macro: { + return super.getFileBlockLocations(p, offset, length); + } + } + + @Override + public synchronized void close() throws IOException Unknown macro: { + super.close(); + AdlFileSystemMetricsSystem.unregisterSource(metricsSourceName); + AdlFileSystemMetricsSystem.fileSystemClosed(); + } + + private String getADLEnabledFeatureSet() Unknown macro: { + // TODO } + + enum StreamState Unknown macro: { + Initial, + DataCachedInLocalBuffer, + StreamEnd + } + + class BatchAppendOutputStream extends OutputStream { + private Path fsPath; + private Param<?, ?>[] parameters; + private byte[] data = null; + private int offset = 0; + private long length = 0; + private boolean eof = false; + private boolean hadError = false; + private int bufferIndex = 0; + private byte[][] dataBuffers = new byte [2] []; + private int bufSize = 0; + private Future<Object> flushTask = null; + + public BatchAppendOutputStream(Path path, int bufferSize, + Param<?, ?>... param) throws IOException { + if (bufferSize < (ADLConfKeys.DEFAULT_BLOCK_SIZE)) Unknown macro: { + bufSize = ADLConfKeys.DEFAULT_BLOCK_SIZE; + } else Unknown macro: { + bufSize = bufferSize; + } + + this.fsPath = path; + this.parameters = param; + this.data = getBuffer(); + FSDataOutputStream createStream = null; + try { + if (featureRedirectOff) { + CreateADLNoRedirectParam skipRedirect = new CreateADLNoRedirectParam( + true); + Param<?, ?>[] tmpParam = featureFlushWhenEOF ? + new Param<?, ?> [param.length + 2] : + new Param<?, ?> [param.length + 1] ; + System.arraycopy(param, 0, tmpParam, 0, param.length); + tmpParam [param.length] = skipRedirect; + if (featureFlushWhenEOF) Unknown macro: { + tmpParam[param.length + 1] = new ADLFlush(false); + } + createStream = new FsPathOutputStreamRunner(ADLPutOpParam.Op.CREATE, + fsPath, 1, tmpParam).run(); + } else Unknown macro: { + createStream = new FsPathOutputStreamRunner(PutOpParam.Op.CREATE, + fsPath, 1, param).run(); + } + } finally { + if (createStream != null) Unknown macro: { + createStream.close(); + } + } + } + + @Override + public final synchronized void write(int b) throws IOException { + if (offset == (data.length)) Unknown macro: { + flush(); + } + + data [offset] = (byte) b; + offset++; + + // Statistics will get incremented again as part of the batch updates, + // decrement here to avoid double value + if (statistics != null) Unknown macro: { + statistics.incrementBytesWritten(-1); + } + } + + @Override + public final synchronized void write(byte[] buf, int off, int len) + throws IOException { + int bytesToWrite = len; + int localOff = off; + int localLen = len; + if (localLen >= data.length) { + // Flush data that is already in our internal buffer + flush(); + + // Keep committing data until we have less than our internal buffers + // length left + do { + try Unknown macro: { + commit(buf, localOff, data.length, eof); + } catch (IOException e) Unknown macro: { + hadError = true; + throw e; + } + localOff += data.length; + localLen -= data.length; + } while (localLen >= data.length); + } + + // At this point, we have less than data.length left to copy from users + // buffer + if (offset + localLen >= data.length) Unknown macro: { + // Users buffer has enough data left to fill our internal buffer + int bytesToCopy = data.length - offset; + System.arraycopy(buf, localOff, data, offset, bytesToCopy); + offset += bytesToCopy; + + // Flush our internal buffer asynchronously + flushAsync(); + localOff += bytesToCopy; + localLen -= bytesToCopy; + } + + if (localLen > 0) Unknown macro: { + // Simply copy the remainder from the users buffer into our internal + // buffer + System.arraycopy(buf, localOff, data, offset, localLen); + offset += localLen; + } + + // Statistics will get incremented again as part of the batch updates, + // decrement here to avoid double value + if (statistics != null) Unknown macro: { + statistics.incrementBytesWritten(-bytesToWrite); + } + instrumentation.rawBytesUploaded(bytesToWrite); + } + + @Override + public final synchronized void flush() throws IOException { + waitForOutstandingFlush(); + if (offset > 0) { + try Unknown macro: { + commit(data, 0, offset, eof); + } catch (IOException e) Unknown macro: { + hadError = true; + throw e; + } + } + + offset = 0; + } + + @Override + public final synchronized void close() throws IOException { + //TODO : 2ns call should not cause any error and no network calls. + if (featureRedirectOff) Unknown macro: { + eof = true; + } + + boolean flushedSomething = false; + if (hadError) Unknown macro: { + // No point proceeding further since the error has occurered and + // stream would be required to upload again. + return; + } else Unknown macro: { + flushedSomething = offset > 0; + flush(); + } + + if (featureRedirectOff) { + // If we didn't flush anything from our internal buffer, we have to + // call the service again + // with an empty payload and flush=true in the url + if (!flushedSomething) Unknown macro: { + commit(null, 0, ADLConfKeys.KB, true); + } + } + + ADLLogger.log(" Total bytes Written : " + (length) + " [" + fsPath + "] "); + } + + private void commit(byte[] buffer, int off, int len, boolean endOfFile) + throws IOException { + OutputStream out = null; + try { + if (featureRedirectOff) { + AppendADLNoRedirectParam skipRedirect = new AppendADLNoRedirectParam( + true); + Param<?, ?>[] tmpParam = featureFlushWhenEOF ? + new Param<?, ?> [parameters.length + 3] : + new Param<?, ?> [parameters.length + 1] ; + System.arraycopy(parameters, 0, tmpParam, 0, parameters.length); + tmpParam [parameters.length] = skipRedirect; + if (featureFlushWhenEOF) Unknown macro: { + tmpParam[parameters.length + 1] = new ADLFlush(endOfFile); + tmpParam[parameters.length + 2] = new OffsetParam(length); + } + + out = new FsPathOutputStreamRunner(ADLPostOpParam.Op.APPEND, fsPath, + len, tmpParam).run(); + } else Unknown macro: { + out = new FsPathOutputStreamRunner(ADLPostOpParam.Op.APPEND, fsPath, + len, parameters).run(); + } + + if (buffer != null) Unknown macro: { + fileStatusCacheManager.remove(fsPath); + out.write(buffer, off, len); + length += len; + } + } finally { + if (out != null) Unknown macro: { + out.close(); + } + } + } + + private void flushAsync() throws IOException { + if (offset > 0) Unknown macro: { + waitForOutstandingFlush(); + + // Submit the new flush task to the executor + flushTask = EXECUTOR.submit(new CommitTask(data, offset, eof)); + + // Get a new internal buffer for the user to write + data = getBuffer(); + offset = 0; + } + } + + private void waitForOutstandingFlush() throws IOException { + if (flushTask != null) { + try Unknown macro: { + flushTask.get(); + } catch (InterruptedException ex) Unknown macro: { + throw new IOException(ex); + } catch (ExecutionException ex) Unknown macro: { + // Wrap the ExecutionException in an IOException for callers can + // only handle IOException + throw new IOException(ex); + } finally Unknown macro: { + flushTask = null; + } + } + } + + private byte[] getBuffer() { + // Switch between the first and second buffer + if (bufferIndex == 0) { + if (dataBuffers [0] == null) Unknown macro: { + dataBuffers[0] = new byte[bufSize]; + } + + bufferIndex = 1; + return dataBuffers [0] ; + } else { + if (dataBuffers [1] == null) Unknown macro: { + dataBuffers[1] = new byte[bufSize]; + } + + bufferIndex = 0; + return dataBuffers [1] ; + } + } + + public class CommitTask implements Callable<Object> { + private byte[] buff; + private int len; + private boolean eof; + + public CommitTask(byte[] buffer, int size, boolean isEnd) Unknown macro: { + buff = buffer; + len = size; + eof = isEnd; + } + + public final Object call() throws IOException Unknown macro: { + commit(buff, 0, len, eof); + return null; + } + } + } + + @SuppressWarnings("checkstyle:javadocmethod") + /** + * Read data from backend in chunks instead of persistent connection. This + * is to avoid slow reader causing socket + * timeout. + */ protected class BatchByteArrayInputStream extends FSInputStream { AF> Formatting. Missing newline above. + + private static final int SIZE4MB = 4 * 1024 * 1024; + private final URL runner; + private volatile byte[] data = null; + private volatile long validDataHoldingSize = 0; + private volatile int bufferOffset = 0; AF> Why volatile here? Needs comments. I have a feeling this is wrong and you need some synchronized blocks below instead. + private volatile long currentFileOffset = 0; + private volatile long nextFileOffset = 0; + private long fileSize = 0; + private String guid; + private StreamState state = StreamState.Initial; + private int maxBufferSize; + private int maxConcurrentConnection; + private Path fsPath; + private boolean streamIsClosed; + private Future[] subtasks = null; + + BatchByteArrayInputStream(URL url, Path p, int bufferSize, + int concurrentConnection) throws IOException { + this.runner = url; + fsPath = p; + FileStatus fStatus = getFileStatus(fsPath); + if (!fStatus.isFile()) Unknown macro: { + throw new IOException("Cannot open the directory " + p + " for " + + "reading"); + } + fileSize = fStatus.getLen(); + guid = getMachineName() + System.currentTimeMillis(); AF> Probably unique. What is impact if this collides? Why not use a random java UUID? + this.maxBufferSize = bufferSize; + this.maxConcurrentConnection = concurrentConnection; + this.streamIsClosed = false; + } + + @Override + public final int read(long position, byte[] buffer, int offset, int length) + throws IOException { + if (streamIsClosed) Unknown macro: { + throw new IOException("Stream already closed"); + } + long oldPos = this.getPos(); + + int nread1; + try Unknown macro: { + this.seek(position); + nread1 = this.read(buffer, offset, length); + } finally Unknown macro: { + this.seek(oldPos); + } + + return nread1; + } + + @Override + public final int read() throws IOException { + if (streamIsClosed) + int status = doBufferAvailabilityCheck(); + if (status == -1) Unknown macro: { + return status; + } + int ch = data [bufferOffset++] & (0xff); AF> this is not thread safe, but looking at FSInputStream, it appears read() is supposed to be. Seems this class is racy in general? + if (statistics != null) Unknown macro: { + statistics.incrementBytesRead(1); + } + return ch; + } + + @Override + public final void readFully(long position, byte[] buffer, int offset, + int length) throws IOException { + if (streamIsClosed) Unknown macro: { + throw new IOException("Stream already closed"); + } + long startTime = System.currentTimeMillis(); + super.readFully(position, buffer, offset, length); + ADLLogger.log("ReadFully1 Time Taken : " + (System.currentTimeMillis() + - startTime)); + if (statistics != null) Unknown macro: { + statistics.incrementBytesRead(length); + } + instrumentation.rawBytesDownloaded(length); + } + + @Override + public final int read(byte[] b, int off, int len) throws IOException { + if (streamIsClosed) + int status = doBufferAvailabilityCheck(); + if (status == -1) + + long startTime = System.currentTimeMillis(); + int byteRead = 0; + long availableBytes = validDataHoldingSize - off; + long requestedBytes = bufferOffset + len - off; + if (requestedBytes <= availableBytes) { + if (b == null) Unknown macro: { + throw new NullPointerException(); + } else if (off < 0 || len < 0 || len > b.length - off) Unknown macro: { + throw new IndexOutOfBoundsException(); + } else if (len == 0) Unknown macro: { + return 0; + } + + ADLLogger.log("AC - [BufferOffset : " + bufferOffset + " " + + "CurrentFileSite : " + currentFileOffset + "] Offset : " + off + + " Length : " + len + " Read Time Taken : " + ( + System.currentTimeMillis() - startTime)); + try Unknown macro: { + System.arraycopy(data, bufferOffset, b, off, len); + } catch (ArrayIndexOutOfBoundsException e) Unknown macro: { + ADLLogger.log("ArrayIndexOutOfBoundsException AC - [BufferOffset } + bufferOffset += len; + byteRead = len; + } else Unknown macro: { + ADLLogger.log("HC - [BufferOffset } + if (statistics != null) Unknown macro: { + statistics.incrementBytesRead(byteRead); + } + instrumentation.rawBytesDownloaded(byteRead); + return byteRead; + } + + private int doBufferAvailabilityCheck() throws IOException { + if (state == StreamState.Initial) Unknown macro: { + ADLLogger.log("Initial Fill"); + validDataHoldingSize = fill(nextFileOffset); + } + + long dataReloadSize = 0; + switch ((int) validDataHoldingSize) { + case -1: + state = StreamState.StreamEnd; + return -1; + case 0: + dataReloadSize = fill(nextFileOffset); + if (dataReloadSize <= 0) Unknown macro: { + state = StreamState.StreamEnd; + return (int) dataReloadSize; + } else Unknown macro: { + validDataHoldingSize = dataReloadSize; + } + break; + default: + break; + } + + if (bufferOffset >= validDataHoldingSize) Unknown macro: { + dataReloadSize = fill(nextFileOffset); + } + + if (bufferOffset >= ((dataReloadSize == 0) ? + validDataHoldingSize : + dataReloadSize)) Unknown macro: { + state = StreamState.StreamEnd; + return -1; + } + + validDataHoldingSize = ((dataReloadSize == 0) ? + validDataHoldingSize : + dataReloadSize); + state = StreamState.DataCachedInLocalBuffer; + return 0; AF> I didn't have time to review this part thoroughly.. Can take better look next round. + } + + public final long fill(final long off) throws IOException { + ADLLogger.log("Fill from " + off); + long startTime = System.currentTimeMillis(); + if (state == StreamState.StreamEnd) Unknown macro: { + return -1; + } + + if (fileSize <= off) + int len = maxBufferSize; + long fileOffset = 0; + boolean isEntireFileCached = true; + if ((fileSize < maxBufferSize)) Unknown macro: { + len = (int) fileSize; + currentFileOffset = 0; + nextFileOffset = 0; + } else { + if (len > (fileSize - off)) Unknown macro: { + len = (int) (fileSize - off); + } + + if (BufferManager.getInstance() + .hasValidDataForOffset(fsPath.toString(), off)) Unknown macro: { + len = (int) ( + BufferManager.getInstance().getBufferOffset() + BufferManager + .getInstance().getBufferSize() - (int) off); + } + + if (len <= 0) Unknown macro: { + len = maxBufferSize; + } + fileOffset = off; + isEntireFileCached = false; + } + + data = null; + BufferManager bm = BufferManager.getInstance(); + data = bm.getEmpty(len); + boolean fetchDataOverNetwork = false; + if (bm.hasData(fsPath.toString(), fileOffset, len)) { + try Unknown macro: { + bm.get(data, fileOffset); + validDataHoldingSize = data.length; + currentFileOffset = fileOffset; + } catch (ArrayIndexOutOfBoundsException e) Unknown macro: { + fetchDataOverNetwork = true; + } + } else Unknown macro: { + fetchDataOverNetwork = true; + } + + if (fetchDataOverNetwork) { + int splitSize = getSplitSize(len); + try Unknown macro: { + validDataHoldingSize = fillDataConcurrently(data, len, fileOffset, + splitSize); + } catch (InterruptedException e) Unknown macro: { + throw new IOException(e.getMessage()); + } + bm.add(data, fsPath.toString(), fileOffset); + currentFileOffset = nextFileOffset; + } + + nextFileOffset += validDataHoldingSize; + state = StreamState.DataCachedInLocalBuffer; + bufferOffset = isEntireFileCached ? (int) off : 0; + ADLLogger.log("Buffer Refill Time Taken : " + (System.currentTimeMillis() + - startTime)); + return validDataHoldingSize; + } + + int getSplitSize(int size) { + if (size <= SIZE4MB) Unknown macro: { + return 1; + } + + // Not practical + if (size > maxBufferSize) Unknown macro: { + size = maxBufferSize; + } + + int equalBufferSplit = size / SIZE4MB; + int splitSize = Math.min(equalBufferSplit, maxConcurrentConnection); + return splitSize; + } + + @Override + public final void seek(long pos) throws IOException { + if (pos == -1) Unknown macro: { + throw new IOException("Bad offset, cannot seek to " + pos); + } + + BufferManager bm = BufferManager.getInstance(); + if (bm.hasValidDataForOffset(fsPath.toString(), pos)) Unknown macro: { + state = StreamState.DataCachedInLocalBuffer; + } else if (pos >= 0) Unknown macro: { + state = StreamState.Initial; + } + + long availableBytes = (currentFileOffset + validDataHoldingSize); + ADLLogger.log("SEEK : " + pos + " Available " + currentFileOffset + " " + + "Count " + availableBytes); + + // Check if this position falls under buffered data + if (pos < currentFileOffset || availableBytes <= 0) Unknown macro: { + validDataHoldingSize = 0; + currentFileOffset = pos; + nextFileOffset = pos; + bufferOffset = 0; + return; + } + + if (pos < availableBytes && pos >= currentFileOffset) Unknown macro: { + state = StreamState.DataCachedInLocalBuffer; + bufferOffset = (int) (pos - currentFileOffset); + } else Unknown macro: { + validDataHoldingSize = 0; + currentFileOffset = pos; + nextFileOffset = pos; + bufferOffset = 0; + } + } + + @Override + public final long getPos() throws IOException { + if (streamIsClosed) Unknown macro: { + throw new IOException("Stream already closed"); + } + return bufferOffset + currentFileOffset; + } + + @Override + public final int available() throws IOException { + if (streamIsClosed) + return Integer.MAX_VALUE; + } + + @Override + public final boolean seekToNewSource(long targetPos) throws IOException Unknown macro: { + return false; + } + + @SuppressWarnings("unchecked") + public final int fillDataConcurrently(byte[] byteArray, int length, + long globalOffset, int splitSize) + throws IOException, InterruptedException { + ADLLogger.log("Fill up Data from " + globalOffset + " len : " + length); + ExecutorService executor = Executors.newFixedThreadPool(splitSize); + subtasks = new Future [splitSize] ; + for (int i = 0; i < splitSize; i++) Unknown macro: { + int offset = i * (length / splitSize); + int splitLength = (splitSize == (i + 1)) ? + (length / splitSize) + (length % splitSize) } + + executor.shutdown(); AF> Is this shutdown() needed? + // wait until all tasks are finished + try Unknown macro: { + executor.awaitTermination(ADLConfKeys.DEFAULT_TIMEOUT_IN_SECONDS, + TimeUnit.SECONDS); AF> Curious, why don't you care about timeout expiring? Can't you just remove this whole try-catch block? The futures are what are ensuring tasks finish below, no? + } catch (InterruptedException e) Unknown macro: { + ADLLogger.log("Interupted } + + int totalBytePainted = 0; + for (int i = 0; i < splitSize; ++i) { + try Unknown macro: { + totalBytePainted += (Integer) subtasks[i].get(); + } catch (InterruptedException e) Unknown macro: { + throw new IOException(e.getCause()); + } catch (ExecutionException e) + } + + if (totalBytePainted != length) Unknown macro: { + throw new IOException("Expected " + length + " bytes, Got " + + totalBytePainted + " bytes"); + } + + return totalBytePainted; + } + + @Override + public final void close() throws IOException Unknown macro: { + BufferManager.getInstance().clear(); + + //need to cleanup the above code the stream and connection close doesnt + // happen here + //flag set to mark close happened, cannot use the stream once closed + streamIsClosed = true; + } + AF> Add function comment w/ params and @return explanation? + private int fillUpData(byte[] buffer, int offset, int length, + long globalOffset) throws IOException { + int totalBytesRead = 0; + final URL offsetUrl = new URL( + runner + "&" + new OffsetParam(String.valueOf(globalOffset)) + "&" + + new LengthParam(String.valueOf(length)) + "&openid=" + guid); AF> Can you explain what this openid does? + HttpURLConnection conn = new URLRunner(GetOpParam.Op.OPEN, offsetUrl, + true).run(); + InputStream in = conn.getInputStream(); + try { + int bytesRead = 0; + while ((bytesRead = in.read(buffer, (int) offset + totalBytesRead, + (int) (length - totalBytesRead))) > 0) Unknown macro: { + totalBytesRead += bytesRead; + } + + // InputStream must be fully consumed to enable http keep-alive + if (bytesRead == 0) { + // Looking for EOF marker byte needs to be read. + if (in.read() != -1) Unknown macro: { + throw new SocketException( + "Server returned more than requested " + "data."); AF> humm.. we expect in.read() to return 0, and then return -1? Also, you can remove the string concatenation in the exception message. + } + } + } finally Unknown macro: { + in.close(); + conn.disconnect(); + } + + return totalBytesRead; + } + + private class BackgroundReadThread implements Callable { + + private final byte[] data; + private int offset; + private int length; + private long globalOffset; + + BackgroundReadThread(byte[] buffer, int off, int size, long position) Unknown macro: { + this.data = buffer; + this.offset = off; + this.length = size; + this.globalOffset = position; + } + + public Object call() throws IOException Unknown macro: { + return fillUpData(data, offset, length, globalOffset); + } + } + } +} \ No newline at end of file diff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java new file mode 100644 index 0000000..a3af0f3 — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateDebugAzureDataLake.java @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one <snip> + * limitations under the License. + * + */ + +package org.apache.hadoop.hdfs.web; AF> As noted before, using filesystem types to acheive logging tags is awkward. Also question why this is not in org.apache.hadoop.fs.adl + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.DelegateToFileSystem; + +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; + +/** + * Use this class implementation to log debug information as part of the + * debug logs. Used during development. + */ +public class PrivateDebugAzureDataLake extends DelegateToFileSystem { <snip> <snip> iff --git hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java new file mode 100644 index 0000000..c3e14f6 — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/resources/ADLFlush.java @@ -0,0 +1,63 @@ +/* <snip> +/** + * Query parameter to notify backend server that the all the data has been + * pushed to over the stream. + * + * Used in operation code Create and Append. + */ +public class ADLFlush extends BooleanParam { + /** AF> I did not review all the HTTP Op Param stuff... someone should. <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md new file mode 100644 index 0000000..4dbc27f — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md @@ -0,0 +1,246 @@ <snip> +## <a name="Introduction" />Introduction + +The hadoop-azure-datalake module provides support for integration with + [Azure Data Lake Store] ( https://azure.microsoft.com/en-in/services/data-lake-store/ ). +The jar file is named azure-datalake-store.jar. + AF> Thank you for the documentation. <snip> +#### <a name="#FileStatus_Cache_Management" />FileStatus Cache Management +Performance is one of the key features of Azure Data Lake Storage service. In order to gain a performance boost, hadoop-azure-datalake module provides basic FileStatus cache management on the client. This reduces the number of REST calls to the backend service. + +FileStatus cache scope is per process and shared between multiple `AdlFileSystem` instances within the process. FileStatus cache is built using ListStatus and GetFileStatus calls to Azure Data Lake Storage. The life of each FileStatus cached object is limited and default is 5 seconds. Time to live FileStatus cached object is configurable through core-site.xml. + +* This is an expermental feature and should be turned off for an unexpected behaviour observed during ListStatus and GetFileStatus operation. * + AF> Seems more than experimental. +To Enable/Disable FileStatus cache management + + <property> + <name>adl.feature.override.cache.filestatus</name> + <value>true</value> <snip> + <property> + <name>dfs.webhdfs.oauth2.refresh.token</name> + <value></value> + </property> + +For ADL FileSystem to take effect. Update core-site.xml with + + <property> + <name>fs.adl.impl</name> + <value>org.apache.hadoop.fs.adl.AdlFileSystem</value> + </property> + + <property> + <name>fs.AbstractFileSystem.adl.impl</name> + <value>org.apache.hadoop.fs.adl.Adl</value> + </property> AF> Again, need to put these in core-default.xml with documentation and defaults, and always use lowercase in the property names. + + +### <a name="Accessing_adl_URLs" />Accessing adl URLs + +After credentials are configured in core-site.xml, any Hadoop component may <snip> diff --git hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java new file mode 100644 index 0000000..1aec96c — /dev/null +++ hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java AF> Thank you for including tests. I did not have time to review the test code this round.
          Hide
          mackrorysd Sean Mackrory added a comment -

          +1 to making the classes in hadoop-azure-datalake all be part of the org.apache.hadoop.fs.adl package. It's hard when reading this code to know if a given class is part of WebHDFS or ADL because so many ADL-specific classes are in org.apache.hadoop.hdfs.web. I don't see any reason for them to be in separate packages.

          Also, to clarify my earlier comment, currently properties like "dfs.webhdfs.oauth2.access.token.provider" are used by both WebHDFS and ADL. This makes it impossible to have WebHDFS and ADL configured for the same client, so use cases like distcp'ing from one file system to another suddenly become unreasonable to support. We should have an overridable prefix in WebHDFS for these properties so ADL can be configured with "dfs.adl.oauth2.access.token.provider", etc. instead.

          Show
          mackrorysd Sean Mackrory added a comment - +1 to making the classes in hadoop-azure-datalake all be part of the org.apache.hadoop.fs.adl package. It's hard when reading this code to know if a given class is part of WebHDFS or ADL because so many ADL-specific classes are in org.apache.hadoop.hdfs.web. I don't see any reason for them to be in separate packages. Also, to clarify my earlier comment, currently properties like "dfs.webhdfs.oauth2.access.token.provider" are used by both WebHDFS and ADL. This makes it impossible to have WebHDFS and ADL configured for the same client, so use cases like distcp'ing from one file system to another suddenly become unreasonable to support. We should have an overridable prefix in WebHDFS for these properties so ADL can be configured with "dfs.adl.oauth2.access.token.provider", etc. instead.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Extracted comments out

          • AF>This class seems to have serious issues that need addressing:
            1. Local race conditions in caller PrivateAzureDataLakeFileSystem
            [Vishwajeet] - Already explained in the previous replies to comments from Chris Nauroth and Lei (Eddy) Xu. In the worst case, there would be a cache miss and this would not break any functionality.

          2. No mechanism for cache invalidation across nodes in the cluster.
          [Vishwajeet] - Scope of cache is per process. Invalidation across nodes would be taken care as part of the distributed cache mechanism in the future.

          • AF>Update comment? This uses "adl" scheme, right?
            [Vishwajeet] - Yes
          • AF>What is this class used for? I didn't see any uses.
            [Vishwajeet] - No longer required. This will is removed.
          • AF>Care to comment why this is in the ..hdfs.web package instead of fs.adl?
            [Vishwajeet] Already explained in the previous comments. In short, requirement to use protected functionality within `WebHdfsFileSystem` implementation to disable redirect operation, Access Runner class for configuration.. etc.
          • AF>?
            [Vishwajeet] Comment was trimmed, updated again.
          • AF>Due to the bug or due to the fix? The fix was merged in 2.8.0, right?
          • AF>I'm not understanding this last sentence, can you explain?

          [Vishwajeet] Not needed anymore since homedirectory would constructed locally and no back-end call would be necessary. Updated comment accordingly.

          • AF>Is this a race condition?
            thread 1> getFileStatus(), cache miss
            super.getStatus -> s1
            cache.get() -> null
            thread 2> delete()
            cache.clear()
            thread 1> cache.put(s1)
            Maybe provide an atomic putIfAbsent() for FileStatusCacheManager. You can
            synchronize on the underlying map object I believe (see
            Collections.synchronizedMap()).

          [Vishwajeet] - Already using `syncMap = Collections.synchronizedMap(map);` We are aware of the limitation of the current implementation. Cache is short lived and would not be persistent effect. Is there a real user scenario where such issue can surface frequently ? We have recommended to turn off the `FileStatus` cache feature in case of misbehavior.

          • AF> Seems like there is a less-likely race condition here. (f is replaced by a directory after checking fs.isFile())
            [Vishwajeet] Why there would be race condition?
          • AF>Similar pattern of get/mutate non-atomically repeats here and below.
            [Vishwajeet] Could you please elaborate the issue ?
          • AF>typo
            [Vishwajeet] Thanks, corrected.
          • AF>Did you guys consider handling this as transparently reconnecting, instead
            of doing separate connections for each op? Seems like performance would be a lot
            better?
            [Vishwajeet] Exactly the same. Since transparent re-connection would require to pass new offset and length value again which is separate op. HTTP persistent connection ensures the same socket is reused so no impact on the performance as well. We have done enough perf test to ensure.
          • AF>I'd expect you to want a connection per-thread, instead of per-op.
            [Vishwajeet] Connection per-op
          • AF> This case could use some perf optimization. e.g. Three calls to get system time. + return fin; + }
            [Vishwajeet] Agree. Corrected.
          • AF> How about adding ADLLogger.logWithTimestamp(). That way, if the logger is disabled, you don't keep getting system time.
            [Vishwajeet] Removed system.currentTimeinMs in case debug or perf flags are OFF.
          • AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations }
            [Vishwajeet] Corrected
          • AF>Just use "name" twice instead of defining host?
            [Vishwajeet] Mainly for readability on block location computation so would not change that.
          • AF> Why the runtime check of a compile-time constant? How about just add a comment near the definition "must be non-zero" + throw new IllegalArgumentException( + "The block size for the given file is not a positive number}
            [Vishwajeet] Removed.
          • AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations }
            [Vishwajeet] Removed
          • AF>Formatting. Missing newline above.
            [Vishwajeet] Using Apache.xml in intellij for formatting per apache guideline. Will look at it later in rulset defined in Apache.xml and raise JIRA accordingly.
          • AF>Why volatile here? Needs comments.
            I have a feeling this is wrong and you need some synchronized blocks below
            instead.
            [Vishwajeet] - We removed synchronization block for performance reason and used Volatile for specific variables which require synchronization. Problem could be when same FsInputSteam instance being used across threads to read from the same file. Couple of variable does not require to be volatile though, updated code.
          • AF>Probably unique. What is impact if this collides? Why not use a random java UUID?
            [Vishwajeet] - For performance reason we have removed UUID usage. Usage is for telemetry purpose. It is used with other parameters to identity the request. For the overlapped value, impact is none.
          • AF>this is not thread safe, but looking at FSInputStream, it appears read() is
            supposed to be.
            Seems this class is racy in general?

          [Vishwajeet] - Assumption is FsInputSteam instance API would not be invoked from multiple threads. Are you aware of such condition which we might encounter?

          • AF>I didn't have time to review this part thoroughly.. Can take
            better look next round.
            [Vishwajeet] Looking forward for your comments.
          • AF>Is this shutdown() needed?
            [Vishwajeet] - Do you see issue any issue here ?
          • AF> Curious, why don't you care about timeout expiring? Can't you just remove this whole try-catch block? The futures are what are ensuring tasks finish below, no? + }
            [Vishwajeet] - Decision is to have time bound operation and if not complete in the duration then fail fast so that application can retry.
          • AF>Add function comment w/ param and @return explanation?
            [Vishwajeet] - Done
          • AF>Can you explain what this openid does?
          • [Vishwajeet] To be used by ADL back-end for telemetry.
          • AF> humm.. we expect in.read() to return 0, and then return -1? Also, you can remove the string concatenation in the exception message. + }
            [Vishwajeet] We encountered this condition when requested n number of bytes match the available n number of bytes however available bytes are (n+1), where +1 is end of network steam marker. Mainly for reusing socket connection. if end of network steam marker is not read then next urlConnection instance re initiate new socket connection to back-end server.
          • AF> As noted before, using filesystem types to achieve logging tags is awkward.
            [Vishwajeet] Additional instrumentation is done in this class for capture perf data hence the separate filesystem exposed. We need to rework on that, i will file separate JIRA for the same.
          • AF>Again, need to put these in core-default.xml with documentation and
            [Vishwajeet] Yes, Will add it as part of the patch. All the configuration defined part of this patch are already lowered case in Path 06, do let me know if you have found any discrepancy?
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Extracted comments out AF>This class seems to have serious issues that need addressing: 1. Local race conditions in caller PrivateAzureDataLakeFileSystem [Vishwajeet] - Already explained in the previous replies to comments from Chris Nauroth and Lei (Eddy) Xu . In the worst case, there would be a cache miss and this would not break any functionality. 2. No mechanism for cache invalidation across nodes in the cluster. [Vishwajeet] - Scope of cache is per process. Invalidation across nodes would be taken care as part of the distributed cache mechanism in the future. AF>Update comment? This uses "adl" scheme, right? [Vishwajeet] - Yes AF>What is this class used for? I didn't see any uses. [Vishwajeet] - No longer required. This will is removed. AF>Care to comment why this is in the ..hdfs.web package instead of fs.adl? [Vishwajeet] Already explained in the previous comments. In short, requirement to use protected functionality within `WebHdfsFileSystem` implementation to disable redirect operation, Access Runner class for configuration.. etc. AF>? [Vishwajeet] Comment was trimmed, updated again. AF>Due to the bug or due to the fix? The fix was merged in 2.8.0, right? AF>I'm not understanding this last sentence, can you explain? [Vishwajeet] Not needed anymore since homedirectory would constructed locally and no back-end call would be necessary. Updated comment accordingly. AF>Is this a race condition? thread 1> getFileStatus(), cache miss super.getStatus -> s1 cache.get() -> null thread 2> delete() cache.clear() thread 1> cache.put(s1) Maybe provide an atomic putIfAbsent() for FileStatusCacheManager. You can synchronize on the underlying map object I believe (see Collections.synchronizedMap()). [Vishwajeet] - Already using `syncMap = Collections.synchronizedMap(map);` We are aware of the limitation of the current implementation. Cache is short lived and would not be persistent effect. Is there a real user scenario where such issue can surface frequently ? We have recommended to turn off the `FileStatus` cache feature in case of misbehavior. AF> Seems like there is a less-likely race condition here. (f is replaced by a directory after checking fs.isFile()) [Vishwajeet] Why there would be race condition? AF>Similar pattern of get/mutate non-atomically repeats here and below. [Vishwajeet] Could you please elaborate the issue ? AF>typo [Vishwajeet] Thanks, corrected. AF>Did you guys consider handling this as transparently reconnecting, instead of doing separate connections for each op? Seems like performance would be a lot better? [Vishwajeet] Exactly the same. Since transparent re-connection would require to pass new offset and length value again which is separate op. HTTP persistent connection ensures the same socket is reused so no impact on the performance as well. We have done enough perf test to ensure. AF>I'd expect you to want a connection per-thread, instead of per-op. [Vishwajeet] Connection per-op AF> This case could use some perf optimization. e.g. Three calls to get system time. + return fin; + } [Vishwajeet] Agree. Corrected. AF> How about adding ADLLogger.logWithTimestamp(). That way, if the logger is disabled, you don't keep getting system time. [Vishwajeet] Removed system.currentTimeinMs in case debug or perf flags are OFF. AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations } [Vishwajeet] Corrected AF>Just use "name" twice instead of defining host? [Vishwajeet] Mainly for readability on block location computation so would not change that. AF> Why the runtime check of a compile-time constant? How about just add a comment near the definition "must be non-zero" + throw new IllegalArgumentException( + "The block size for the given file is not a positive number} [Vishwajeet] Removed. AF> Redundant check of isLogEnabled() + ADLLogger.log("getFileBlockLocations } [Vishwajeet] Removed AF>Formatting. Missing newline above. [Vishwajeet] Using Apache.xml in intellij for formatting per apache guideline. Will look at it later in rulset defined in Apache.xml and raise JIRA accordingly. AF>Why volatile here? Needs comments. I have a feeling this is wrong and you need some synchronized blocks below instead. [Vishwajeet] - We removed synchronization block for performance reason and used Volatile for specific variables which require synchronization. Problem could be when same FsInputSteam instance being used across threads to read from the same file. Couple of variable does not require to be volatile though, updated code. AF>Probably unique. What is impact if this collides? Why not use a random java UUID? [Vishwajeet] - For performance reason we have removed UUID usage. Usage is for telemetry purpose. It is used with other parameters to identity the request. For the overlapped value, impact is none. AF>this is not thread safe, but looking at FSInputStream, it appears read() is supposed to be. Seems this class is racy in general? [Vishwajeet] - Assumption is FsInputSteam instance API would not be invoked from multiple threads. Are you aware of such condition which we might encounter? AF>I didn't have time to review this part thoroughly.. Can take better look next round. [Vishwajeet] Looking forward for your comments. AF>Is this shutdown() needed? [Vishwajeet] - Do you see issue any issue here ? AF> Curious, why don't you care about timeout expiring? Can't you just remove this whole try-catch block? The futures are what are ensuring tasks finish below, no? + } [Vishwajeet] - Decision is to have time bound operation and if not complete in the duration then fail fast so that application can retry. AF>Add function comment w/ param and @return explanation? [Vishwajeet] - Done AF>Can you explain what this openid does? [Vishwajeet] To be used by ADL back-end for telemetry. AF> humm.. we expect in.read() to return 0, and then return -1? Also, you can remove the string concatenation in the exception message. + } [Vishwajeet] We encountered this condition when requested n number of bytes match the available n number of bytes however available bytes are (n+1), where +1 is end of network steam marker. Mainly for reusing socket connection. if end of network steam marker is not read then next urlConnection instance re initiate new socket connection to back-end server. AF> As noted before, using filesystem types to achieve logging tags is awkward. [Vishwajeet] Additional instrumentation is done in this class for capture perf data hence the separate filesystem exposed. We need to rework on that, i will file separate JIRA for the same. AF>Again, need to put these in core-default.xml with documentation and [Vishwajeet] Yes, Will add it as part of the patch. All the configuration defined part of this patch are already lowered case in Path 06, do let me know if you have found any discrepancy?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Sorry Sean Mackrory i missed your earlier comment.

          For `org.apache.hadoop.hdfs.web` packaging - Already explained in the previous replies. Short summary is current design of `WebHdfsFileSystem` does not allow certain configuration to be done outside `org.apache.hadoop.hdfs.web` namespace.

          For common properties like dfs.webhdfs.oauth2.access.token.provider - The scope of the this patch set does not cover the design changes on the existing OAuth2 implementation in ASF. You have bring a valid point and we need to create separate JIRA for the same. I will take it once this patch is through.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Sorry Sean Mackrory i missed your earlier comment. For `org.apache.hadoop.hdfs.web` packaging - Already explained in the previous replies. Short summary is current design of `WebHdfsFileSystem` does not allow certain configuration to be done outside `org.apache.hadoop.hdfs.web` namespace. For common properties like dfs.webhdfs.oauth2.access.token.provider - The scope of the this patch set does not cover the design changes on the existing OAuth2 implementation in ASF. You have bring a valid point and we need to create separate JIRA for the same. I will take it once this patch is through.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          For the common concern over the dependency on `org.apache.hadoop.hdfs.web` packaging - Already explained in the previous replies. However i would like to reiterate that due to current design constraint in `org.apache.hadoop.hdfs.web` namespace, extended file system from `WebHdfsFileSystem` can not access certain functionalities outside `org.apache.hadoop.hdfs.web`. Example : Control over additional or existing query parameters, HTTP configuration .. etc. Being said that, We do desire to have only `org.apache.hadoop.fs.adl` package which contains all the functionalities.

          In order to achieve our common goal, I would have to file few more JIRA's on the `org.apache.hadoop.hdfs.web` package and work on to make extended FileSystem from `org.apache.hadoop.hdfs.web` configurable and refactor existing ADL package accordingly. I would take up this activity once the Rev 1 i.e. this patch set is pushed in to ASF.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - For the common concern over the dependency on `org.apache.hadoop.hdfs.web` packaging - Already explained in the previous replies. However i would like to reiterate that due to current design constraint in `org.apache.hadoop.hdfs.web` namespace, extended file system from `WebHdfsFileSystem` can not access certain functionalities outside `org.apache.hadoop.hdfs.web`. Example : Control over additional or existing query parameters, HTTP configuration .. etc. Being said that, We do desire to have only `org.apache.hadoop.fs.adl` package which contains all the functionalities. In order to achieve our common goal, I would have to file few more JIRA's on the `org.apache.hadoop.hdfs.web` package and work on to make extended FileSystem from `org.apache.hadoop.hdfs.web` configurable and refactor existing ADL package accordingly. I would take up this activity once the Rev 1 i.e. this patch set is pushed in to ASF.
          Hide
          fabbri Aaron Fabbri added a comment -

          Vishwajeet Dusane thanks for the responses. It seems you are not convinced on some of the synchronization bugs I pointed out.

          Two hints:

          • Understand why ConcurrentHashMap adds functions like putIfAbsent(). Just because get() and put() are synchronized, does not make this safe:
          if (map.get() == null) {
              Thing t = new Thing();
              map.put(t);
          }
          
          • Use of volatile. Given:
                volatile byte[] data = null;
                volatile int bufferOffset = 0;
            

          This does not make code like this thread safe:

          int read() {
              return data[bufferOffset++] & 0xff;
          }
          

          The argument "we probably don't need thread safety anyways" implies you should just remove all synchronization. If not needed, it would hurt performance.

          If I'm wrong on any of this please call it out. Thank you.

          Show
          fabbri Aaron Fabbri added a comment - Vishwajeet Dusane thanks for the responses. It seems you are not convinced on some of the synchronization bugs I pointed out. Two hints: Understand why ConcurrentHashMap adds functions like putIfAbsent(). Just because get() and put() are synchronized, does not make this safe: if (map.get() == null) { Thing t = new Thing(); map.put(t); } Use of volatile. Given: volatile byte[] data = null; volatile int bufferOffset = 0; This does not make code like this thread safe: int read() { return data[bufferOffset++] & 0xff; } The argument "we probably don't need thread safety anyways" implies you should just remove all synchronization. If not needed, it would hurt performance. If I'm wrong on any of this please call it out. Thank you.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Aaron Fabbri Thanks a lot for your comments.

          For FileStatus Cache - I agree on the race condition situations. My question about your concern is, could it break any functionality in such a situation? and I think it would not break any common functionality. Based on the variety of Hadoop applications we have executed with this code.

          So let me try to break down the discussion based on the scenarios.

            • What is FileStatus Cache?
              • FileStatus cache is simple process level cache which mirrors backend storage FileStatus objects.
              • Time to live on the FileStatus cached object is limited. 5 seconds default and configurable through core-site.xml
              • FileStatus objects are stored in Synchronized LinkedHashMap. Where key is fully qualified file path and value is FileStatus java object along with time to live information.
              • FileStatus cache is built based on successful responses to GetFileStatus and ListStatus calls for existing files/folders. Non existent files/folder are not maintained in the cache.
              • FileStatus cache motivation is to avoid multiple GetFileStatus calls to the ADL backend and as a result gain better performance for job startup and during execution.
                I will try to break down in to some scenarios that may occur.
            • Scenario 1 : Concurrent get request for the same FileStatus object
              • Multiple threads trying to access same FileStatus object.
                Example: GetFileStatus call for path /a.txt from multiple threads within process when FileStatus instance present in the cache.
              • Should not be a problem, Valid FileStatus object is returned to caller across threads.
            • Scenario 2 : Concurrent put request for the same FileStatus object
              • Multiple threads updating same FileStatus object.
                	public String thread1()
                	{
                	    // FileStatus fileStatus - For storage filepath /a.txt 
                		...
                		fileStatusCacheManager.put(fileStatus,5); // Race condition
                		...
                	}
                	...
                	public String thread2()
                	{
                	    // FileStatus fileStatus - For storage file /a.txt 
                		...
                		fileStatusCacheManager.put(fileStatus,5); // Race condition
                		...
                	}
                	
              • Whoever wins the race, Metadata for FileStatus instance would be constant for the same file /a.txt
              • Hence the latest and greatest value for /a.txt is valid value anyway.
            • Scenario 3 : Concurrent get/put request for the same FileStatus object
              	public String thread1()
              	{
              	    // FileStatus fileStatus - For storage filepath /a.txt 
              		...
              		fileStatusCacheManager.put(fileStatus,5); // Race condition
              		...
              	}
              	...
              	public String thread2()
              	{
              	    Path f = new Path("/a.txt");
              		...
              		FileStatus fileStatus = fileStatusCacheManager.get(makeQualified(f)); // Race condition
              		...
              	}
              	
              • Depending upon order of execution thread2 may or may not get latest value updated from thread1. Even synchronization of blocks are not going to guarantee that.
              • Worst case thread2 gets NULL i.e. FileStatus object for /a.txt does not exist in the cache so thread2 would fall back to invoke ADL backend call to GetFileStatus.
              • Does not break any functionality in this case as well.
            • Scenario 4: Concurrent get/remove request for the same FileStatus object
              	public String thread1()
              	{
              	    Path f = new Path("/a.txt");
              		...
              		fileStatusCacheManager.remove(makeQualified(f)); // Cache cleanup caused due to delete/rename/Create operation on /a.txt. Race condition
              		...
              	}
              	...
              	public String thread2()
              	{
              	    Path f = new Path("/a.txt");
              		...
              		FileStatus fileStatus = fileStatusCacheManager.get(makeQualified(f)); // Race condition
              		...
              	}
              	
              • Depending upon order of execution thread2 may get stale information from the cache. Similar to the above scenario, synchronization of blocks are not going to solve this either
              • Unavoidable situation with/without FileStatus cache and with/without ADL storage backend.
            • Scenario 5: Concurrent put/remove request for the different FileStatus object
              	public String thread1()
              	{
              	    Path f = new Path("/a.txt");
              		...
              		fileStatusCacheManager.remove(makeQualified(f)); // Cache cleanup caused due to delete/rename/Create operation on /a.txt. Race condition
              		...
              	}
              	...
              	public String thread2()
              	{
              	    // FileStatus fileStatus - For storage filepath /a.txt 
              		...
              		fileStatusCacheManager.put(fileStatus,5); // Race condition
              		...
              	}
              	
              • Depending upon order of execution, FileStatus cache may hold a stale instance for 5 seconds. Similar to above, synchronization of blocks are not going to solve this either.
              • This is a corner case and may involve misbehavior to the application, based on there use case. In such situation FileStatus cache should be turned off.
          For volatile usage - Totally agree with you. Like i mentioned in the earlier comment, i will remove volatile usage for those variables.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Aaron Fabbri Thanks a lot for your comments. For FileStatus Cache - I agree on the race condition situations. My question about your concern is, could it break any functionality in such a situation? and I think it would not break any common functionality. Based on the variety of Hadoop applications we have executed with this code. So let me try to break down the discussion based on the scenarios. What is FileStatus Cache? FileStatus cache is simple process level cache which mirrors backend storage FileStatus objects. Time to live on the FileStatus cached object is limited. 5 seconds default and configurable through core-site.xml FileStatus objects are stored in Synchronized LinkedHashMap. Where key is fully qualified file path and value is FileStatus java object along with time to live information. FileStatus cache is built based on successful responses to GetFileStatus and ListStatus calls for existing files/folders. Non existent files/folder are not maintained in the cache. FileStatus cache motivation is to avoid multiple GetFileStatus calls to the ADL backend and as a result gain better performance for job startup and during execution. I will try to break down in to some scenarios that may occur. Scenario 1 : Concurrent get request for the same FileStatus object Multiple threads trying to access same FileStatus object. Example: GetFileStatus call for path /a.txt from multiple threads within process when FileStatus instance present in the cache. Should not be a problem, Valid FileStatus object is returned to caller across threads. Scenario 2 : Concurrent put request for the same FileStatus object Multiple threads updating same FileStatus object. public String thread1() { // FileStatus fileStatus - For storage filepath /a.txt ... fileStatusCacheManager.put(fileStatus,5); // Race condition ... } ... public String thread2() { // FileStatus fileStatus - For storage file /a.txt ... fileStatusCacheManager.put(fileStatus,5); // Race condition ... } Whoever wins the race, Metadata for FileStatus instance would be constant for the same file /a.txt Hence the latest and greatest value for /a.txt is valid value anyway. Scenario 3 : Concurrent get/put request for the same FileStatus object public String thread1() { // FileStatus fileStatus - For storage filepath /a.txt ... fileStatusCacheManager.put(fileStatus,5); // Race condition ... } ... public String thread2() { Path f = new Path( "/a.txt" ); ... FileStatus fileStatus = fileStatusCacheManager.get(makeQualified(f)); // Race condition ... } Depending upon order of execution thread2 may or may not get latest value updated from thread1. Even synchronization of blocks are not going to guarantee that. Worst case thread2 gets NULL i.e. FileStatus object for /a.txt does not exist in the cache so thread2 would fall back to invoke ADL backend call to GetFileStatus. Does not break any functionality in this case as well. Scenario 4: Concurrent get/remove request for the same FileStatus object public String thread1() { Path f = new Path( "/a.txt" ); ... fileStatusCacheManager.remove(makeQualified(f)); // Cache cleanup caused due to delete/rename/Create operation on /a.txt. Race condition ... } ... public String thread2() { Path f = new Path( "/a.txt" ); ... FileStatus fileStatus = fileStatusCacheManager.get(makeQualified(f)); // Race condition ... } Depending upon order of execution thread2 may get stale information from the cache. Similar to the above scenario, synchronization of blocks are not going to solve this either Unavoidable situation with/without FileStatus cache and with/without ADL storage backend. Scenario 5: Concurrent put/remove request for the different FileStatus object public String thread1() { Path f = new Path( "/a.txt" ); ... fileStatusCacheManager.remove(makeQualified(f)); // Cache cleanup caused due to delete/rename/Create operation on /a.txt. Race condition ... } ... public String thread2() { // FileStatus fileStatus - For storage filepath /a.txt ... fileStatusCacheManager.put(fileStatus,5); // Race condition ... } Depending upon order of execution, FileStatus cache may hold a stale instance for 5 seconds. Similar to above, synchronization of blocks are not going to solve this either. This is a corner case and may involve misbehavior to the application, based on there use case. In such situation FileStatus cache should be turned off. For volatile usage - Totally agree with you. Like i mentioned in the earlier comment, i will remove volatile usage for those variables.
          Hide
          cnauroth Chris Nauroth added a comment -

          If people would find it easier for line-by-line commentary, we could import the patch into ReviewBoard (reviews.apache.org) or a GitHub pull request. Reviewers, please comment if you'd like to go that way.

          Vishwajeet Dusane, thank you for updating the patch in response to feedback. Here are a few more comments on patch v006.

          1. pom.xml should use hadoop-project as its parent. See hadoop-tools/hadoop-azure/pom.xml for an example.
          2. artifactId should be hadoop-azure-datalake. The resulting built jar will then be hadoop-azure-datalake-<VERSION>.jar. This naming will be consistent with the other Hadoop build artifacts.
          3. Please indent pom.xml by 2 spaces.
          4. Thank you for the documentation!
          5. Would you also please update hadoop-project/src/site/site.xml, so that we get a hyperlink in the left nav? Look for the "Hadoop Compatible File Systems" section, and you can add a link similar to the one for WASB.
          6. Typo: "File relication factor"
          7. Typo: Azure Data Lake Storage is access path syntax is
          8. Typo: expermental
          9. There are several links to this URL: https://azure.microsoft.com/en-in/services/data-lake-store/ . Perhaps it's just my opinion, but the content there strays a little too much towards marketecture. The readers of our documentation tend to be a technical audience that is already using Hadoop, so they want to find usage information and technical details as quickly as possible. What do you think about linking to this other page instead, which seems to be the "developer hub"? https://azure.microsoft.com/en-in/documentation/services/data-lake-store/
          10. I see widespread mishandling of InterruptedException throughout the patch. The classic article on how to handle InterruptedException is here: http://www.ibm.com/developerworks/library/j-jtp05236/ . The bottom line is that it shouldn't be swallowed or rethrown as a different exception type unless you first restore the thread's interrupted status by calling Thread.currentThread().interrupt(). Failure to restore interrupted status can cause unexpected behavior at other layers of the code that are expecting to observe the interrupted status later, such as during shutdown handling. There is a lot of existing Hadoop code that doesn't do this correctly, so reading Hadoop code isn't always a good example. Let's not introduce more of the same problems in new code though.

          Now for some of the broader discussion points:

          We do have extended Contact test cases which integrated with back-end service however those test are not pushed as part of this check in. I will create separate JIRA for the Live mode test cases.

          I would strongly prefer to have the contract tests in place from day one as part of this patch. Is that possible?

          We have seen multiple times that the contract test suites have been valuable for identifying subtle bugs in file system semantics. It's a much more effective way to find and fix these kinds of bugs compared to getting multiple hours into a complex system integration test run with higher-level applications, only to have it fail because of some file system edge case.

          Addiionally, it's really vital to have some kind of integration test suite available that really connects to the back-end service instead of using mocks. We have seen several times that mock-based testing has failed to expose bugs that surface when integrating with the real service. Let's try to run the contract tests against the real service, not the mock. The recent addition of the contract tests in WASB in HADOOP-12535 is a good example of how to achieve this.

          The tests you already have in the patch are still useful, so please keep them. The good thing about the mock-based tests is that they'll continue to run on Jenkins pre-commit when people post new patches.

          Reason behind hiding logging through was to switch quickly between Log and System.out.println during debugging. Quickest way is change the code than configuration file. We will migrate to use SLF4J but not part of this patch release. is that fine?

          Maybe I'm missing something, but I'm having a hard time justifying this. Log4J makes it easy to reroute its logging to stdout via configuration using the ConsoleAppender. Managing it through configuration instead of code removes the latency of the compile/deploy cycle when someone decides to change logging settings. Hadoop developers are familiar and comfortable with Log4J configuration, so if they want to debug, their first instinct is going to be to change Log4J configuration, not change code. Can we please switch to SLF4J and drop the ADLLogger class now instead of deferring it to later?

          ADLConfKeys#LOG_VERSION is to capture code instrumentation version. this information is used only during debugging session.

          Can you please elaborate on what LOG_VERSION is supposed to mean? Does "code instrumentation version" mean that this is trying to capture the version of the Hadoop software that is running, or is there some special significance to the hard-coded "1.2.1" value? If it's meant to indicate the Hadoop software version, then the Hadoop VersionInfo class I mentioned would be a good fit.

          How do Hadoop folks feel about this hadoop-tools/hadoop-azure-datalake code declaring classes in the hadoop.hdfs.web package? I feel it needs cleanup.

          Yes, I agree with Aaron Fabbri. Generally, it's a poor practice for multiple jars to bundle classes in the same Java package. This can cause classpath confusion. For example, if two jars accidentally introduce a Foo class in the same Java package, then the JVM might not load the one you thought it would load at runtime. There are places in the Hadoop codebase where we have had to do this, but that was mostly for backwards-compatibility reasons, not a desirable design choice.

          Also, to clarify my earlier comment, currently properties like "dfs.webhdfs.oauth2.access.token.provider" are used by both WebHDFS and ADL. This makes it impossible to have WebHDFS and ADL configured for the same client, so use cases like distcp'ing from one file system to another suddenly become unreasonable to support.

          Yes, I agree with Sean Mackrory. This is important, because using DistCp for external backup to a cloud provider is a common use case. WebHDFS (backed by HDFS) is often recommended for DistCp due to its cross-version compatibility story, so it's important for both ADL and WebHDFS backed by HDFS to co-exist in the same deployment.

          Assumption is FsInputSteam instance API would not be invoked from multiple threads. Are you aware of such condition which we might encounter?

          Off the top of my head, I can't point out a specific application that performs multi-threaded reads on a single shared input stream, but I'm pretty sure this happens somewhere. However, I know for sure that HDFS does provide thread-safety for this usage pattern, as do WASB and S3. By failing to match the semantics of HDFS, this does create a potential bug. In general, I agree with the comments from Aaron Fabbri about race conditions.

          We removed synchronization block for performance reason and used Volatile for specific variables which require synchronization. Problem could be when same FsInputSteam instance being used across threads to read from the same file. Couple of variable does not require to be volatile though, updated code.

          It sounds like you attempted coarser synchronization at one point but weren't satisfied with the performance. The challenge will be to provide thread-safety without compromising on performance.

          In order to achieve our common goal, I would have to file few more JIRA's on the `org.apache.hadoop.hdfs.web` package and work on to make extended FileSystem from `org.apache.hadoop.hdfs.web` configurable and refactor existing ADL package accordingly. I would take up this activity once the Rev 1 i.e. this patch set is pushed in to ASF.

          Do you have a sense of how much work is required in org.apache.hadoop.hdfs.web to set up this patch so that all of its classes can reside in org.apache.hadoop.fs.adl? Maybe we could get that done now as a set of pre-requisite patches before going ahead with this one.

          Show
          cnauroth Chris Nauroth added a comment - If people would find it easier for line-by-line commentary, we could import the patch into ReviewBoard (reviews.apache.org) or a GitHub pull request. Reviewers, please comment if you'd like to go that way. Vishwajeet Dusane , thank you for updating the patch in response to feedback. Here are a few more comments on patch v006. pom.xml should use hadoop-project as its parent. See hadoop-tools/hadoop-azure/pom.xml for an example. artifactId should be hadoop-azure-datalake. The resulting built jar will then be hadoop-azure-datalake-<VERSION>.jar. This naming will be consistent with the other Hadoop build artifacts. Please indent pom.xml by 2 spaces. Thank you for the documentation! Would you also please update hadoop-project/src/site/site.xml, so that we get a hyperlink in the left nav? Look for the "Hadoop Compatible File Systems" section, and you can add a link similar to the one for WASB. Typo: "File relication factor" Typo: Azure Data Lake Storage is access path syntax is Typo: expermental There are several links to this URL: https://azure.microsoft.com/en-in/services/data-lake-store/ . Perhaps it's just my opinion, but the content there strays a little too much towards marketecture . The readers of our documentation tend to be a technical audience that is already using Hadoop, so they want to find usage information and technical details as quickly as possible. What do you think about linking to this other page instead, which seems to be the "developer hub"? https://azure.microsoft.com/en-in/documentation/services/data-lake-store/ I see widespread mishandling of InterruptedException throughout the patch. The classic article on how to handle InterruptedException is here: http://www.ibm.com/developerworks/library/j-jtp05236/ . The bottom line is that it shouldn't be swallowed or rethrown as a different exception type unless you first restore the thread's interrupted status by calling Thread.currentThread().interrupt() . Failure to restore interrupted status can cause unexpected behavior at other layers of the code that are expecting to observe the interrupted status later, such as during shutdown handling. There is a lot of existing Hadoop code that doesn't do this correctly, so reading Hadoop code isn't always a good example. Let's not introduce more of the same problems in new code though. Now for some of the broader discussion points: We do have extended Contact test cases which integrated with back-end service however those test are not pushed as part of this check in. I will create separate JIRA for the Live mode test cases. I would strongly prefer to have the contract tests in place from day one as part of this patch. Is that possible? We have seen multiple times that the contract test suites have been valuable for identifying subtle bugs in file system semantics. It's a much more effective way to find and fix these kinds of bugs compared to getting multiple hours into a complex system integration test run with higher-level applications, only to have it fail because of some file system edge case. Addiionally, it's really vital to have some kind of integration test suite available that really connects to the back-end service instead of using mocks. We have seen several times that mock-based testing has failed to expose bugs that surface when integrating with the real service. Let's try to run the contract tests against the real service, not the mock. The recent addition of the contract tests in WASB in HADOOP-12535 is a good example of how to achieve this. The tests you already have in the patch are still useful, so please keep them. The good thing about the mock-based tests is that they'll continue to run on Jenkins pre-commit when people post new patches. Reason behind hiding logging through was to switch quickly between Log and System.out.println during debugging. Quickest way is change the code than configuration file. We will migrate to use SLF4J but not part of this patch release. is that fine? Maybe I'm missing something, but I'm having a hard time justifying this. Log4J makes it easy to reroute its logging to stdout via configuration using the ConsoleAppender. Managing it through configuration instead of code removes the latency of the compile/deploy cycle when someone decides to change logging settings. Hadoop developers are familiar and comfortable with Log4J configuration, so if they want to debug, their first instinct is going to be to change Log4J configuration, not change code. Can we please switch to SLF4J and drop the ADLLogger class now instead of deferring it to later? ADLConfKeys#LOG_VERSION is to capture code instrumentation version. this information is used only during debugging session. Can you please elaborate on what LOG_VERSION is supposed to mean? Does "code instrumentation version" mean that this is trying to capture the version of the Hadoop software that is running, or is there some special significance to the hard-coded "1.2.1" value? If it's meant to indicate the Hadoop software version, then the Hadoop VersionInfo class I mentioned would be a good fit. How do Hadoop folks feel about this hadoop-tools/hadoop-azure-datalake code declaring classes in the hadoop.hdfs.web package? I feel it needs cleanup. Yes, I agree with Aaron Fabbri . Generally, it's a poor practice for multiple jars to bundle classes in the same Java package. This can cause classpath confusion. For example, if two jars accidentally introduce a Foo class in the same Java package, then the JVM might not load the one you thought it would load at runtime. There are places in the Hadoop codebase where we have had to do this, but that was mostly for backwards-compatibility reasons, not a desirable design choice. Also, to clarify my earlier comment, currently properties like "dfs.webhdfs.oauth2.access.token.provider" are used by both WebHDFS and ADL. This makes it impossible to have WebHDFS and ADL configured for the same client, so use cases like distcp'ing from one file system to another suddenly become unreasonable to support. Yes, I agree with Sean Mackrory . This is important, because using DistCp for external backup to a cloud provider is a common use case. WebHDFS (backed by HDFS) is often recommended for DistCp due to its cross-version compatibility story, so it's important for both ADL and WebHDFS backed by HDFS to co-exist in the same deployment. Assumption is FsInputSteam instance API would not be invoked from multiple threads. Are you aware of such condition which we might encounter? Off the top of my head, I can't point out a specific application that performs multi-threaded reads on a single shared input stream, but I'm pretty sure this happens somewhere. However, I know for sure that HDFS does provide thread-safety for this usage pattern, as do WASB and S3. By failing to match the semantics of HDFS, this does create a potential bug. In general, I agree with the comments from Aaron Fabbri about race conditions. We removed synchronization block for performance reason and used Volatile for specific variables which require synchronization. Problem could be when same FsInputSteam instance being used across threads to read from the same file. Couple of variable does not require to be volatile though, updated code. It sounds like you attempted coarser synchronization at one point but weren't satisfied with the performance. The challenge will be to provide thread-safety without compromising on performance. In order to achieve our common goal, I would have to file few more JIRA's on the `org.apache.hadoop.hdfs.web` package and work on to make extended FileSystem from `org.apache.hadoop.hdfs.web` configurable and refactor existing ADL package accordingly. I would take up this activity once the Rev 1 i.e. this patch set is pushed in to ASF. Do you have a sense of how much work is required in org.apache.hadoop.hdfs.web to set up this patch so that all of its classes can reside in org.apache.hadoop.fs.adl ? Maybe we could get that done now as a set of pre-requisite patches before going ahead with this one.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thank you Chris Nauroth. I am not going to respond on the comments which i agree with you (And i will incorporate them in the next patch upload along with comments from Aaron Fabbri) and appreciate the detail review.

          • For contract test cases - Will get back on this later.
          • LOG_VERSION is telemetry information used in Adl back-end only.
          • For packaging namespace - https://reviews.apache.org/r/44169/ has the change that Adl requires to not use 'org.apache.hadoop.hdfs.web' and reside in org.apache.hadoop.fs.adl only. aaron fabbri - I could not find your user name in the https://reviews.apache.org/r/44169/ so please have a look.
          • For Segregation of common properties - Like i mentioned in the earlier comment, can this not be taken up in separate JIRA? or do you (Sean Mackrory and Chris Douglas) expect it to be taken care as part of this patch only since OAuth2 is being used by Adl only so far?
          • Thread-safety on FsInputSteam - Will get back on this later.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thank you Chris Nauroth . I am not going to respond on the comments which i agree with you (And i will incorporate them in the next patch upload along with comments from Aaron Fabbri ) and appreciate the detail review. For contract test cases - Will get back on this later. LOG_VERSION is telemetry information used in Adl back-end only. For packaging namespace - https://reviews.apache.org/r/44169/ has the change that Adl requires to not use 'org.apache.hadoop.hdfs.web' and reside in org.apache.hadoop.fs.adl only. aaron fabbri - I could not find your user name in the https://reviews.apache.org/r/44169/ so please have a look. For Segregation of common properties - Like i mentioned in the earlier comment, can this not be taken up in separate JIRA? or do you ( Sean Mackrory and Chris Douglas ) expect it to be taken care as part of this patch only since OAuth2 is being used by Adl only so far? Thread-safety on FsInputSteam - Will get back on this later.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          From the code review comments, there were 5 major issues that we need to call out, we have listed below the actions we have taken to address those comments

          • Synchronization during read stream – Though the use case seems to be unusual to allow concurrent access on stream. Accepted comment to have synchronized blocks across. Synchronized blocks are causing additional Avg. 7ms latency during read operation.
          • Patch size too large & inclusion of Live test cases – we have Split the patch into multiple JIRAs
            • HADOOP-12666 - core updates
            • HADOOP-12875 - Test updates including mechanism for live updates.
            • HADOOP-12876 - file metadata cache management
            • Separate JIRAs for Telemetry and instrumentation related updates would be raised once we agree and commit Hadoop-12666
          • FileStatus Cache management – Raised separate JIRA HADOOP-12876 to cover specific discussion around the cache.
          • Package namespace (Remove dependency from org.apache.hadoop.hdfs.web) – In order to remove dependency from org.apache.hadoop.hdfs.web package. One of the proposal (https://reviews.apache.org/r/44169/)is to modify access level for the dependent parts in org.apache.hadoop.hdfs.web package to public.
          • Allow Webhdfs and Adl file system to coexist (Common configuration parameter like dfs.webhdfs.oauth2.refresh.token ) – As of today only Adl is compliant to oauth2 protocol. To support only adl specific configuration requires changes in ASF code. We would like to take this as separate change set than to cover as part of this change set.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - From the code review comments, there were 5 major issues that we need to call out, we have listed below the actions we have taken to address those comments Synchronization during read stream – Though the use case seems to be unusual to allow concurrent access on stream. Accepted comment to have synchronized blocks across. Synchronized blocks are causing additional Avg. 7ms latency during read operation. Patch size too large & inclusion of Live test cases – we have Split the patch into multiple JIRAs HADOOP-12666 - core updates HADOOP-12875 - Test updates including mechanism for live updates. HADOOP-12876 - file metadata cache management Separate JIRAs for Telemetry and instrumentation related updates would be raised once we agree and commit Hadoop-12666 FileStatus Cache management – Raised separate JIRA HADOOP-12876 to cover specific discussion around the cache. Package namespace (Remove dependency from org.apache.hadoop.hdfs.web) – In order to remove dependency from org.apache.hadoop.hdfs.web package. One of the proposal ( https://reviews.apache.org/r/44169/)is to modify access level for the dependent parts in org.apache.hadoop.hdfs.web package to public. Allow Webhdfs and Adl file system to coexist (Common configuration parameter like dfs.webhdfs.oauth2.refresh.token ) – As of today only Adl is compliant to oauth2 protocol. To support only adl specific configuration requires changes in ASF code. We would like to take this as separate change set than to cover as part of this change set.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 0s Maven dependency ordering for branch
          +1 mvninstall 6m 38s trunk passed
          +1 compile 6m 17s trunk passed with JDK v1.8.0_74
          +1 compile 6m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 3m 6s trunk passed
          +1 mvneclipse 1m 1s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 1m 56s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 18s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 30s the patch passed
          +1 compile 6m 8s the patch passed with JDK v1.8.0_74
          +1 javac 6m 8s the patch passed
          +1 compile 6m 50s the patch passed with JDK v1.7.0_95
          +1 javac 6m 50s the patch passed
          -1 checkstyle 1m 4s root: patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 3m 27s the patch passed
          +1 mvneclipse 1m 18s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 xml 0m 2s The patch has 1 ill-formed XML file(s).
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 22s the patch passed
          +1 javadoc 2m 9s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 38s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74.
          -1 unit 16m 3s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74.
          -1 unit 3m 47s hadoop-tools in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 9s hadoop-tools-dist in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_95.
          -1 unit 16m 10s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 12s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          -1 unit 4m 14s hadoop-tools in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 11s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 22s Patch generated 4 ASF License warnings.
          103m 25s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.security.TestJNIGroupsMapping
            hadoop.io.file.tfile.TestTFileSeqFileComparison
            hadoop.fs.TestLocalFSFileContextCreateMkdir
            hadoop.fs.shell.TestPathData
            hadoop.io.compress.bzip2.TestBzip2CompressorDecompressor
            hadoop.io.TestSequenceFileAppend
            hadoop.fs.contract.localfs.TestLocalFSContractLoaded
            hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
            hadoop.fs.shell.TestAclCommands
            hadoop.ha.TestHealthMonitor
            hadoop.ipc.TestMultipleProtocolServer
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.io.serializer.TestSerializationFactory
            hadoop.security.authorize.TestProxyServers
            hadoop.fs.contract.ftp.TestFTPContractOpen
            hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem
            hadoop.crypto.TestCryptoStreams
            hadoop.util.TestGenericsUtil
            hadoop.crypto.key.TestKeyProvider
            hadoop.ipc.TestSocketFactory
            hadoop.io.TestSecureIOUtils
            hadoop.io.file.tfile.TestTFile
            hadoop.fs.contract.ftp.TestFTPContractCreate
            hadoop.http.TestHttpServerWebapps
            hadoop.crypto.TestCryptoCodec
            hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.fs.viewfs.TestFcMainOperationsLocalFs
            hadoop.fs.TestLocalDirAllocator
            hadoop.security.TestCompositeGroupMapping
            hadoop.fs.contract.rawlocal.TestRawlocalContractSeek
            hadoop.ipc.TestRPC
            hadoop.fs.contract.localfs.TestLocalFSContractGetFileStatus
            hadoop.http.TestGlobalFilter
            hadoop.fs.shell.TestLs
            hadoop.util.TestFindClass
            hadoop.security.alias.TestCredentialProviderFactory
            hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
            hadoop.http.TestHttpServerLifecycle
            hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider
            hadoop.net.TestStaticMapping
            hadoop.fs.shell.find.TestName
            hadoop.fs.TestFileContext
            hadoop.ha.TestNodeFencer
            hadoop.crypto.TestCryptoStreamsForLocalFS
            hadoop.fs.TestDelegateToFileSystem
            hadoop.net.TestScriptBasedMapping
            hadoop.security.ssl.TestSSLFactory
            hadoop.net.TestSwitchMapping
            hadoop.security.TestHttpCrossOriginFilterInitializer
            hadoop.conf.TestConfigurationSubclass
            hadoop.fs.TestTruncatedInputBug
            hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir
            hadoop.http.TestAuthenticationSessionCookie
            hadoop.fs.TestGetFileBlockLocations
            hadoop.fs.viewfs.TestFcPermissionsLocalFs
            hadoop.security.TestDoAsEffectiveUser
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.fs.TestLocalFSFileContextMainOperations
            hadoop.fs.viewfs.TestViewFsURIs
            hadoop.crypto.key.TestKeyProviderDelegationTokenExtension
            hadoop.ipc.TestMiniRPCBenchmark
            hadoop.fs.TestStat
            hadoop.security.TestLdapGroupsMapping
            hadoop.io.TestSequenceFileSerialization
            hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
            hadoop.fs.TestSymlinkLocalFSFileContext
            hadoop.http.TestHttpCookieFlag
            hadoop.log.TestLogLevel
            hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
            hadoop.http.TestPathFilter
            hadoop.ipc.TestSaslRPC
            hadoop.io.TestMapFile
            hadoop.util.TestNativeCrc32
            hadoop.io.file.tfile.TestVLong
            hadoop.service.TestServiceLifecycle
            hadoop.fs.TestLocalFileSystemPermission
            hadoop.fs.TestPath
            hadoop.fs.viewfs.TestChRootedFs
            hadoop.crypto.random.TestOsSecureRandom
            hadoop.util.TestGenericOptionsParser
            hadoop.io.file.tfile.TestTFileStreams
            hadoop.fs.contract.localfs.TestLocalFSContractMkdir
            hadoop.fs.contract.localfs.TestLocalFSContractRename
            hadoop.fs.TestFcLocalFsPermission
            hadoop.ha.TestSshFenceByTcpPort
            hadoop.fs.TestFileUtil
            hadoop.io.TestSequenceFile
            hadoop.ipc.TestWeightedRoundRobinMultiplexer
            hadoop.io.serializer.avro.TestAvroSerialization
            hadoop.ha.TestShellCommandFencer
            hadoop.security.authorize.TestServiceAuthorization
            hadoop.fs.permission.TestFsPermission
            hadoop.fs.TestS3_LocalFileContextURI
            hadoop.crypto.TestCryptoStreamsWithJceAesCtrCryptoCodec
            hadoop.fs.viewfs.TestViewfsFileStatus
            hadoop.fs.contract.rawlocal.TestRawlocalContractRename
            hadoop.io.compress.TestCodecPool
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.fs.contract.localfs.TestLocalFSContractSeek
            hadoop.security.TestShellBasedIdMapping
            hadoop.ipc.TestProtoBufRpc
            hadoop.http.TestHttpServerLogs
            hadoop.fs.shell.TestXAttrCommands
            hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
            hadoop.ipc.TestRPCServerShutdown
            hadoop.fs.contract.localfs.TestLocalFSContractDelete
            hadoop.io.TestWritable
            hadoop.security.TestKDiagNoKDC
            hadoop.conf.TestReconfiguration
            hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
            hadoop.cli.TestCLI
            hadoop.security.token.delegation.web.TestWebDelegationToken
            hadoop.fs.shell.TestCount
            hadoop.fs.shell.find.TestIname
            hadoop.io.erasurecode.coder.TestHHXORErasureCoder
            hadoop.http.TestServletFilter
            hadoop.ipc.TestProtoBufRPCCompatibility
            hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestFsShellReturnCode
            hadoop.fs.viewfs.TestViewFsConfig
            hadoop.fs.TestFcLocalFsUtil
            hadoop.io.file.tfile.TestTFileNoneCodecsStreams
            hadoop.conf.TestConfServlet
            hadoop.ha.TestZKFailoverController
            hadoop.metrics2.source.TestJvmMetrics
            hadoop.io.file.tfile.TestTFileSplit
            hadoop.io.erasurecode.coder.TestRSErasureCoder
            hadoop.ipc.TestIPC
            hadoop.security.TestGroupFallback
            hadoop.fs.shell.find.TestPrint0
            hadoop.io.TestBloomMapFile
            hadoop.net.TestNetUtils
            hadoop.crypto.TestCryptoStreamsNormal
            hadoop.ipc.TestIdentityProviders
            hadoop.io.TestGenericWritable
            hadoop.fs.TestFsShellCopy
            hadoop.io.file.tfile.TestTFileComparator2
            hadoop.fs.TestFileSystemCanonicalization
            hadoop.fs.shell.TestCopy
            hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
            hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
            hadoop.ipc.TestServer
            hadoop.http.TestSSLHttpServer
            hadoop.fs.contract.rawlocal.TestRawlocalContractDelete
            hadoop.fs.contract.localfs.TestLocalFSContractOpen
            hadoop.io.file.tfile.TestTFileSeek
            hadoop.security.TestKDiag
            hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport
            hadoop.io.compress.zlib.TestZlibCompressorDecompressor
            hadoop.fs.TestChecksumFileSystem
            hadoop.ipc.TestRPCWaitForProxy
            hadoop.security.token.delegation.web.TestDelegationTokenManager
            hadoop.fs.shell.find.TestFind
            hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
            hadoop.io.TestSequenceFileSync
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.contract.ftp.TestFTPContractMkdir
            hadoop.ha.TestFailoverController
            hadoop.fs.viewfs.TestChRootedFileSystem
            hadoop.security.authorize.TestProxyUsers
            hadoop.security.alias.TestCredShell
            hadoop.fs.shell.TestMove
            hadoop.io.compress.TestCompressorDecompressor
            hadoop.metrics2.sink.TestRollingFileSystemSink
            hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays
            hadoop.fs.TestSymlinkLocalFSFileSystem
            hadoop.ipc.TestIPCServerResponder
            hadoop.util.TestNativeLibraryChecker
            hadoop.security.TestLdapGroupsMappingWithPosixGroup
            hadoop.fs.TestFSMainOperationsLocalFileSystem
            hadoop.fs.contract.localfs.TestLocalFSContractAppend
            hadoop.security.TestUGILoginFromKeytab
            hadoop.fs.contract.localfs.TestLocalFSContractSetTimes
            hadoop.security.TestSecurityUtil
            hadoop.net.TestTableMapping
            hadoop.fs.TestLocalFsFCStatistics
            hadoop.fs.TestFileSystemInitialization
            hadoop.fs.contract.rawlocal.TestRawlocalContractSetTimes
            hadoop.util.hash.TestHash
            hadoop.io.TestDefaultStringifier
            hadoop.fs.contract.ftp.TestFTPContractDelete
            hadoop.io.serializer.TestWritableSerialization
            hadoop.net.TestScriptBasedMappingWithDependency
            hadoop.io.TestArrayFile
            hadoop.io.compress.TestCodecFactory
            hadoop.security.TestWhitelistBasedResolver
            hadoop.fs.TestFsShell
            hadoop.fs.TestHarFileSystemBasics
            hadoop.ipc.TestFairCallQueue
            hadoop.security.authorize.TestAccessControlList
            hadoop.fs.contract.ftp.TestFTPContractRename
            hadoop.fs.TestFileContextResolveAfs
            hadoop.io.file.tfile.TestTFileComparators
            hadoop.fs.TestListFiles
            hadoop.security.TestProxyUserFromEnv
            hadoop.ipc.TestRPCCallBenchmark
            hadoop.fs.shell.TestTextCommand
            hadoop.io.file.tfile.TestTFileLzoCodecsStreams
            hadoop.fs.TestHarFileSystem
            hadoop.ha.TestHAAdmin
            hadoop.fs.contract.localfs.TestLocalFSContractCreate
            hadoop.io.retry.TestDefaultRetryPolicy
            hadoop.ipc.TestRPCCompatibility
            hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
            hadoop.conf.TestGetInstances
            hadoop.fs.viewfs.TestViewFileSystemDelegation
            hadoop.fs.viewfs.TestViewFsTrash
            hadoop.io.file.tfile.TestTFileUnsortedByteArrays
            hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress
            hadoop.io.compress.TestCodec
            hadoop.security.TestGroupsCaching
            hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus
            hadoop.fs.TestLocalFileSystem
            hadoop.tracing.TestTraceUtils
            hadoop.fs.TestLocal_S3FileContextURI
            hadoop.crypto.key.TestKeyShell
            hadoop.security.TestAuthenticationFilter
            hadoop.util.TestDiskChecker
            hadoop.conf.TestDeprecatedKeys
            hadoop.fs.viewfs.TestViewFsLocalFs
            hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
            hadoop.fs.sftp.TestSFTPFileSystem
            hadoop.security.TestUserFromEnv
            hadoop.crypto.key.TestKeyProviderFactory
            hadoop.io.compress.TestCompressionStreamReuse
            hadoop.conf.TestConfigurationDeprecation
            hadoop.io.TestSetFile
            hadoop.crypto.key.TestKeyProviderCryptoExtension
            hadoop.conf.TestConfiguration
            hadoop.util.TestNodeHealthScriptRunner
            hadoop.fs.viewfs.TestFcCreateMkdirLocalFs
            hadoop.fs.TestFileContextDeleteOnExit
            hadoop.io.file.tfile.TestTFileByteArrays
            hadoop.jmx.TestJMXJsonServlet
            hadoop.http.lib.TestStaticUserWebFilter
            hadoop.http.TestHttpServer
            hadoop.fs.TestTrash
            hadoop.fs.TestDU
            hadoop.fs.TestAvroFSInput
            hadoop.security.TestUserGroupInformation
            hadoop.tools.TestHadoopArchives
            hadoop.tools.TestDistCpSync
            hadoop.tools.mapred.TestCopyCommitter
            hadoop.tools.TestDistCpWithXAttrs
            hadoop.tools.TestDistCpWithRawXAttrs
            hadoop.tools.TestIntegration
            hadoop.tools.mapred.TestCopyOutputFormat
            hadoop.tools.mapred.TestUniformSizeInputFormat
            hadoop.tools.TestOptionsParser
            hadoop.tools.TestGlobbedCopyListing
            hadoop.tools.mapred.lib.TestDynamicInputFormat
            hadoop.tools.mapred.TestCopyMapper
            hadoop.tools.TestDistCpSystem
            hadoop.tools.TestDistCpViewFs
            hadoop.tools.TestFileBasedCopyListing
            hadoop.tools.TestCopyListing
            hadoop.tools.TestExternalCall
            hadoop.tools.util.TestDistCpUtils
            hadoop.tools.mapred.TestRetriableFileCopyCommand
            hadoop.tools.TestDistCpWithAcls
            hadoop.mapred.gridmix.TestFileQueue
            hadoop.mapred.gridmix.TestGridMixClasses
            hadoop.mapred.gridmix.TestHighRamJob
            hadoop.mapred.gridmix.TestPseudoLocalFs
            hadoop.mapred.gridmix.TestDistCacheEmulation
            hadoop.mapred.gridmix.TestSleepJob
            hadoop.mapred.gridmix.TestUserResolve
            hadoop.mapred.gridmix.TestCompressionEmulationUtils
            hadoop.mapred.gridmix.TestFilePool
            hadoop.mapred.gridmix.TestRecordFactory
            hadoop.mapred.gridmix.TestGridmixSummary
            hadoop.mapred.gridmix.TestLoadJob
            hadoop.mapred.gridmix.TestGridmixSubmission
            hadoop.mapred.gridmix.TestResourceUsageEmulators
            hadoop.mapred.gridmix.TestGridmixMemoryEmulation
            hadoop.tools.TestDistCh
            hadoop.fs.azure.TestNativeAzureFileSystemMocked
            hadoop.fs.azure.TestBlobMetadata
            hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
            hadoop.fs.azure.contract.TestAzureNativeContractMkdir
            hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
            hadoop.fs.azure.TestOutOfBandAzureBlobOperations
            hadoop.fs.azure.contract.TestAzureNativeContractRename
            hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
            hadoop.fs.azure.TestNativeAzureFileSystemLive
            hadoop.fs.azure.TestFileSystemOperationExceptionHandling
            hadoop.fs.azure.TestNativeAzureFSPageBlobLive
            hadoop.fs.azure.TestWasbUriAndConfiguration
            hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
            hadoop.fs.azure.TestWasbFsck
            hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
            hadoop.fs.azure.TestNativeAzureFileSystemContractLive
            hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
            hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
            hadoop.fs.azure.contract.TestAzureNativeContractDelete
            hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
            hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
            hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
            hadoop.fs.azure.TestNativeAzureFileSystemAppend
            hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
            hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
            hadoop.fs.azure.contract.TestAzureNativeContractOpen
            hadoop.fs.azure.TestBlobDataValidation
            hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
            hadoop.fs.azure.contract.TestAzureNativeContractAppend
            hadoop.fs.azure.contract.TestAzureNativeContractSeek
            hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
            hadoop.fs.azure.TestBlobTypeSpeedDifference
            hadoop.fs.azure.TestContainerChecks
            hadoop.fs.azure.TestAzureFileSystemErrorConditions
            hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
            hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
            hadoop.fs.azure.contract.TestAzureNativeContractCreate
            hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
            hadoop.streaming.TestStreamReduceNone
            hadoop.streaming.TestStreamingBackground
            hadoop.streaming.TestStreamXmlRecordReader
            hadoop.streaming.TestUnconsumedInput
            hadoop.streaming.TestStreamingCounters
            hadoop.streaming.TestStreamingOutputOnlyKeys
            hadoop.streaming.TestTypedBytesStreaming
            hadoop.streaming.TestStreamingExitStatus
            hadoop.streaming.TestDumpTypedBytes
            hadoop.streaming.io.TestKeyOnlyTextOutputReader
            hadoop.streaming.TestStreamAggregate
            hadoop.streaming.TestMRFramework
            hadoop.streaming.TestMultipleArchiveFiles
            hadoop.streaming.TestStreaming
            hadoop.streaming.TestStreamingKeyValue
            hadoop.streaming.TestStreamJob
            hadoop.streaming.TestStreamingFailure
            hadoop.streaming.TestStreamingCombiner
            hadoop.streaming.TestStreamingStderr
            hadoop.streaming.TestStreamDataProtocol
            hadoop.streaming.TestStreamXmlMultipleRecords
            hadoop.streaming.TestAutoInputFormat
            hadoop.streaming.mapreduce.TestStreamXmlRecordReader
            hadoop.streaming.TestStreamingBadRecords
            hadoop.streaming.TestStreamingSeparator
            hadoop.streaming.TestSymLink
            hadoop.streaming.TestStreamingOutputKeyValueTypes
            hadoop.streaming.TestMultipleCachefiles
            hadoop.streaming.TestGzipInput
            hadoop.streaming.TestLoadTypedBytes
            hadoop.streaming.TestRawBytesStreaming
            hadoop.streaming.TestFileArgs
            hadoop.contrib.utils.join.TestDataJoin
            hadoop.yarn.sls.appmaster.TestAMSimulator
            hadoop.yarn.sls.nodemanager.TestNMSimulator
            hadoop.yarn.sls.TestSLSRunner
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.fs.TestFileSystemCaching
            org.apache.hadoop.io.nativeio.TestNativeIO
          JDK v1.7.0_95 Failed junit tests hadoop.security.TestJNIGroupsMapping
            hadoop.io.file.tfile.TestTFileSeqFileComparison
            hadoop.fs.TestLocalFSFileContextCreateMkdir
            hadoop.fs.shell.TestPathData
            hadoop.io.compress.bzip2.TestBzip2CompressorDecompressor
            hadoop.io.TestSequenceFileAppend
            hadoop.fs.contract.localfs.TestLocalFSContractLoaded
            hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
            hadoop.fs.shell.TestAclCommands
            hadoop.ha.TestHealthMonitor
            hadoop.ipc.TestMultipleProtocolServer
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.io.serializer.TestSerializationFactory
            hadoop.security.authorize.TestProxyServers
            hadoop.fs.contract.ftp.TestFTPContractOpen
            hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem
            hadoop.crypto.TestCryptoStreams
            hadoop.util.TestGenericsUtil
            hadoop.crypto.key.TestKeyProvider
            hadoop.ipc.TestSocketFactory
            hadoop.io.TestSecureIOUtils
            hadoop.io.file.tfile.TestTFile
            hadoop.fs.contract.ftp.TestFTPContractCreate
            hadoop.http.TestHttpServerWebapps
            hadoop.crypto.TestCryptoCodec
            hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.fs.viewfs.TestFcMainOperationsLocalFs
            hadoop.fs.TestLocalDirAllocator
            hadoop.security.TestCompositeGroupMapping
            hadoop.fs.contract.rawlocal.TestRawlocalContractSeek
            hadoop.ipc.TestRPC
            hadoop.fs.contract.localfs.TestLocalFSContractGetFileStatus
            hadoop.http.TestGlobalFilter
            hadoop.fs.shell.TestLs
            hadoop.util.TestFindClass
            hadoop.security.alias.TestCredentialProviderFactory
            hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
            hadoop.http.TestHttpServerLifecycle
            hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider
            hadoop.net.TestStaticMapping
            hadoop.fs.shell.find.TestName
            hadoop.fs.TestFileContext
            hadoop.ha.TestNodeFencer
            hadoop.crypto.TestCryptoStreamsForLocalFS
            hadoop.fs.TestDelegateToFileSystem
            hadoop.net.TestScriptBasedMapping
            hadoop.security.ssl.TestSSLFactory
            hadoop.net.TestSwitchMapping
            hadoop.security.TestHttpCrossOriginFilterInitializer
            hadoop.conf.TestConfigurationSubclass
            hadoop.fs.TestTruncatedInputBug
            hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir
            hadoop.http.TestAuthenticationSessionCookie
            hadoop.fs.TestGetFileBlockLocations
            hadoop.fs.viewfs.TestFcPermissionsLocalFs
            hadoop.security.TestDoAsEffectiveUser
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.fs.TestLocalFSFileContextMainOperations
            hadoop.fs.viewfs.TestViewFsURIs
            hadoop.crypto.key.TestKeyProviderDelegationTokenExtension
            hadoop.ipc.TestMiniRPCBenchmark
            hadoop.fs.TestStat
            hadoop.security.TestLdapGroupsMapping
            hadoop.io.TestSequenceFileSerialization
            hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
            hadoop.fs.TestSymlinkLocalFSFileContext
            hadoop.http.TestHttpCookieFlag
            hadoop.log.TestLogLevel
            hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
            hadoop.http.TestPathFilter
            hadoop.ipc.TestSaslRPC
            hadoop.io.TestMapFile
            hadoop.util.TestNativeCrc32
            hadoop.io.file.tfile.TestVLong
            hadoop.service.TestServiceLifecycle
            hadoop.fs.TestLocalFileSystemPermission
            hadoop.fs.TestPath
            hadoop.fs.viewfs.TestChRootedFs
            hadoop.crypto.random.TestOsSecureRandom
            hadoop.util.TestGenericOptionsParser
            hadoop.io.file.tfile.TestTFileStreams
            hadoop.fs.contract.localfs.TestLocalFSContractMkdir
            hadoop.fs.contract.localfs.TestLocalFSContractRename
            hadoop.fs.TestFcLocalFsPermission
            hadoop.ha.TestSshFenceByTcpPort
            hadoop.fs.TestFileUtil
            hadoop.io.TestSequenceFile
            hadoop.ipc.TestWeightedRoundRobinMultiplexer
            hadoop.io.serializer.avro.TestAvroSerialization
            hadoop.ha.TestShellCommandFencer
            hadoop.security.authorize.TestServiceAuthorization
            hadoop.fs.permission.TestFsPermission
            hadoop.fs.TestS3_LocalFileContextURI
            hadoop.crypto.TestCryptoStreamsWithJceAesCtrCryptoCodec
            hadoop.fs.viewfs.TestViewfsFileStatus
            hadoop.fs.contract.rawlocal.TestRawlocalContractRename
            hadoop.io.compress.TestCodecPool
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.fs.contract.localfs.TestLocalFSContractSeek
            hadoop.security.TestShellBasedIdMapping
            hadoop.ipc.TestProtoBufRpc
            hadoop.http.TestHttpServerLogs
            hadoop.fs.shell.TestXAttrCommands
            hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
            hadoop.ipc.TestRPCServerShutdown
            hadoop.fs.contract.localfs.TestLocalFSContractDelete
            hadoop.io.TestWritable
            hadoop.security.TestKDiagNoKDC
            hadoop.conf.TestReconfiguration
            hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
            hadoop.cli.TestCLI
            hadoop.security.token.delegation.web.TestWebDelegationToken
            hadoop.fs.shell.TestCount
            hadoop.fs.shell.find.TestIname
            hadoop.io.erasurecode.coder.TestHHXORErasureCoder
            hadoop.http.TestServletFilter
            hadoop.ipc.TestProtoBufRPCCompatibility
            hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestFsShellReturnCode
            hadoop.fs.viewfs.TestViewFsConfig
            hadoop.fs.TestFcLocalFsUtil
            hadoop.io.file.tfile.TestTFileNoneCodecsStreams
            hadoop.conf.TestConfServlet
            hadoop.ha.TestZKFailoverController
            hadoop.metrics2.source.TestJvmMetrics
            hadoop.io.file.tfile.TestTFileSplit
            hadoop.io.erasurecode.coder.TestRSErasureCoder
            hadoop.ipc.TestIPC
            hadoop.security.TestGroupFallback
            hadoop.fs.shell.find.TestPrint0
            hadoop.io.TestBloomMapFile
            hadoop.net.TestNetUtils
            hadoop.crypto.TestCryptoStreamsNormal
            hadoop.ipc.TestIdentityProviders
            hadoop.io.TestGenericWritable
            hadoop.fs.TestFsShellCopy
            hadoop.io.file.tfile.TestTFileComparator2
            hadoop.fs.TestFileSystemCanonicalization
            hadoop.fs.shell.TestCopy
            hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
            hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
            hadoop.ipc.TestServer
            hadoop.http.TestSSLHttpServer
            hadoop.fs.contract.rawlocal.TestRawlocalContractDelete
            hadoop.fs.contract.localfs.TestLocalFSContractOpen
            hadoop.io.file.tfile.TestTFileSeek
            hadoop.security.TestKDiag
            hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport
            hadoop.io.compress.zlib.TestZlibCompressorDecompressor
            hadoop.fs.TestChecksumFileSystem
            hadoop.ipc.TestRPCWaitForProxy
            hadoop.security.token.delegation.web.TestDelegationTokenManager
            hadoop.fs.shell.find.TestFind
            hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
            hadoop.io.TestSequenceFileSync
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.contract.ftp.TestFTPContractMkdir
            hadoop.ha.TestFailoverController
            hadoop.fs.viewfs.TestChRootedFileSystem
            hadoop.security.authorize.TestProxyUsers
            hadoop.security.alias.TestCredShell
            hadoop.fs.shell.TestMove
            hadoop.io.compress.TestCompressorDecompressor
            hadoop.metrics2.sink.TestRollingFileSystemSink
            hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays
            hadoop.fs.TestSymlinkLocalFSFileSystem
            hadoop.ipc.TestIPCServerResponder
            hadoop.util.TestNativeLibraryChecker
            hadoop.security.TestLdapGroupsMappingWithPosixGroup
            hadoop.fs.TestFSMainOperationsLocalFileSystem
            hadoop.fs.contract.localfs.TestLocalFSContractAppend
            hadoop.security.TestUGILoginFromKeytab
            hadoop.fs.contract.localfs.TestLocalFSContractSetTimes
            hadoop.security.TestSecurityUtil
            hadoop.net.TestTableMapping
            hadoop.fs.TestLocalFsFCStatistics
            hadoop.fs.TestFileSystemInitialization
            hadoop.fs.contract.rawlocal.TestRawlocalContractSetTimes
            hadoop.util.hash.TestHash
            hadoop.io.TestDefaultStringifier
            hadoop.fs.contract.ftp.TestFTPContractDelete
            hadoop.io.serializer.TestWritableSerialization
            hadoop.net.TestScriptBasedMappingWithDependency
            hadoop.io.TestArrayFile
            hadoop.io.compress.TestCodecFactory
            hadoop.security.TestWhitelistBasedResolver
            hadoop.fs.TestFsShell
            hadoop.fs.TestHarFileSystemBasics
            hadoop.ipc.TestFairCallQueue
            hadoop.security.authorize.TestAccessControlList
            hadoop.fs.contract.ftp.TestFTPContractRename
            hadoop.fs.TestFileContextResolveAfs
            hadoop.io.file.tfile.TestTFileComparators
            hadoop.fs.TestListFiles
            hadoop.security.TestProxyUserFromEnv
            hadoop.ipc.TestRPCCallBenchmark
            hadoop.fs.shell.TestTextCommand
            hadoop.io.file.tfile.TestTFileLzoCodecsStreams
            hadoop.fs.TestHarFileSystem
            hadoop.ha.TestHAAdmin
            hadoop.fs.contract.localfs.TestLocalFSContractCreate
            hadoop.io.retry.TestDefaultRetryPolicy
            hadoop.ipc.TestRPCCompatibility
            hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
            hadoop.conf.TestGetInstances
            hadoop.fs.viewfs.TestViewFileSystemDelegation
            hadoop.fs.viewfs.TestViewFsTrash
            hadoop.io.file.tfile.TestTFileUnsortedByteArrays
            hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress
            hadoop.io.compress.TestCodec
            hadoop.security.TestGroupsCaching
            hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus
            hadoop.fs.TestLocalFileSystem
            hadoop.tracing.TestTraceUtils
            hadoop.fs.TestLocal_S3FileContextURI
            hadoop.crypto.key.TestKeyShell
            hadoop.security.TestAuthenticationFilter
            hadoop.util.TestDiskChecker
            hadoop.conf.TestDeprecatedKeys
            hadoop.fs.viewfs.TestViewFsLocalFs
            hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
            hadoop.fs.sftp.TestSFTPFileSystem
            hadoop.security.TestUserFromEnv
            hadoop.crypto.key.TestKeyProviderFactory
            hadoop.io.compress.TestCompressionStreamReuse
            hadoop.conf.TestConfigurationDeprecation
            hadoop.io.TestSetFile
            hadoop.crypto.key.TestKeyProviderCryptoExtension
            hadoop.conf.TestConfiguration
            hadoop.util.TestNodeHealthScriptRunner
            hadoop.fs.viewfs.TestFcCreateMkdirLocalFs
            hadoop.fs.TestFileContextDeleteOnExit
            hadoop.io.file.tfile.TestTFileByteArrays
            hadoop.jmx.TestJMXJsonServlet
            hadoop.http.lib.TestStaticUserWebFilter
            hadoop.http.TestHttpServer
            hadoop.fs.TestTrash
            hadoop.fs.TestDU
            hadoop.fs.TestAvroFSInput
            hadoop.security.TestUserGroupInformation
            hadoop.tools.TestHadoopArchives
            hadoop.tools.TestDistCpSync
            hadoop.tools.mapred.TestCopyCommitter
            hadoop.tools.TestDistCpWithXAttrs
            hadoop.tools.TestDistCpWithRawXAttrs
            hadoop.tools.TestIntegration
            hadoop.tools.mapred.TestCopyOutputFormat
            hadoop.tools.mapred.TestUniformSizeInputFormat
            hadoop.tools.TestOptionsParser
            hadoop.tools.TestGlobbedCopyListing
            hadoop.tools.mapred.lib.TestDynamicInputFormat
            hadoop.tools.mapred.TestCopyMapper
            hadoop.tools.TestDistCpSystem
            hadoop.tools.TestDistCpViewFs
            hadoop.tools.TestFileBasedCopyListing
            hadoop.tools.TestCopyListing
            hadoop.tools.TestExternalCall
            hadoop.tools.util.TestDistCpUtils
            hadoop.tools.mapred.TestRetriableFileCopyCommand
            hadoop.tools.TestDistCpWithAcls
            hadoop.mapred.gridmix.TestFileQueue
            hadoop.mapred.gridmix.TestGridMixClasses
            hadoop.mapred.gridmix.TestHighRamJob
            hadoop.mapred.gridmix.TestPseudoLocalFs
            hadoop.mapred.gridmix.TestDistCacheEmulation
            hadoop.mapred.gridmix.TestSleepJob
            hadoop.mapred.gridmix.TestUserResolve
            hadoop.mapred.gridmix.TestCompressionEmulationUtils
            hadoop.mapred.gridmix.TestFilePool
            hadoop.mapred.gridmix.TestRecordFactory
            hadoop.mapred.gridmix.TestGridmixSummary
            hadoop.mapred.gridmix.TestLoadJob
            hadoop.mapred.gridmix.TestGridmixSubmission
            hadoop.mapred.gridmix.TestResourceUsageEmulators
            hadoop.mapred.gridmix.TestGridmixMemoryEmulation
            hadoop.tools.TestDistCh
            hadoop.fs.azure.TestNativeAzureFileSystemMocked
            hadoop.fs.azure.TestBlobMetadata
            hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
            hadoop.fs.azure.contract.TestAzureNativeContractMkdir
            hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
            hadoop.fs.azure.TestOutOfBandAzureBlobOperations
            hadoop.fs.azure.contract.TestAzureNativeContractRename
            hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
            hadoop.fs.azure.TestNativeAzureFileSystemLive
            hadoop.fs.azure.TestFileSystemOperationExceptionHandling
            hadoop.fs.azure.TestNativeAzureFSPageBlobLive
            hadoop.fs.azure.TestWasbUriAndConfiguration
            hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
            hadoop.fs.azure.TestWasbFsck
            hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
            hadoop.fs.azure.TestNativeAzureFileSystemContractLive
            hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
            hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
            hadoop.fs.azure.contract.TestAzureNativeContractDelete
            hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
            hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
            hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
            hadoop.fs.azure.TestNativeAzureFileSystemAppend
            hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
            hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
            hadoop.fs.azure.contract.TestAzureNativeContractOpen
            hadoop.fs.azure.TestBlobDataValidation
            hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
            hadoop.fs.azure.contract.TestAzureNativeContractAppend
            hadoop.fs.azure.contract.TestAzureNativeContractSeek
            hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
            hadoop.fs.azure.TestBlobTypeSpeedDifference
            hadoop.fs.azure.TestContainerChecks
            hadoop.fs.azure.TestAzureFileSystemErrorConditions
            hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
            hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
            hadoop.fs.azure.contract.TestAzureNativeContractCreate
            hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
            hadoop.streaming.TestStreamReduceNone
            hadoop.streaming.TestStreamingBackground
            hadoop.streaming.TestStreamXmlRecordReader
            hadoop.streaming.TestUnconsumedInput
            hadoop.streaming.TestStreamingCounters
            hadoop.streaming.TestStreamingOutputOnlyKeys
            hadoop.streaming.TestTypedBytesStreaming
            hadoop.streaming.TestStreamingExitStatus
            hadoop.streaming.TestDumpTypedBytes
            hadoop.streaming.io.TestKeyOnlyTextOutputReader
            hadoop.streaming.TestStreamAggregate
            hadoop.streaming.TestMRFramework
            hadoop.streaming.TestMultipleArchiveFiles
            hadoop.streaming.TestStreaming
            hadoop.streaming.TestStreamingKeyValue
            hadoop.streaming.TestStreamJob
            hadoop.streaming.TestStreamingFailure
            hadoop.streaming.TestStreamingCombiner
            hadoop.streaming.TestStreamingStderr
            hadoop.streaming.TestStreamDataProtocol
            hadoop.streaming.TestStreamXmlMultipleRecords
            hadoop.streaming.TestAutoInputFormat
            hadoop.streaming.mapreduce.TestStreamXmlRecordReader
            hadoop.streaming.TestStreamingBadRecords
            hadoop.streaming.TestStreamingSeparator
            hadoop.streaming.TestSymLink
            hadoop.streaming.TestStreamingOutputKeyValueTypes
            hadoop.streaming.TestMultipleCachefiles
            hadoop.streaming.TestGzipInput
            hadoop.streaming.TestLoadTypedBytes
            hadoop.streaming.TestRawBytesStreaming
            hadoop.streaming.TestFileArgs
            hadoop.contrib.utils.join.TestDataJoin
            hadoop.yarn.sls.appmaster.TestAMSimulator
            hadoop.yarn.sls.nodemanager.TestNMSimulator
            hadoop.yarn.sls.TestSLSRunner
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.fs.TestFileSystemCaching
            org.apache.hadoop.io.nativeio.TestNativeIO



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791749/HADOOP-12666-007.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ea081d554c13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fd1c09b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/diff-checkstyle-root.txt
          xml https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/xml.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 0s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 6m 17s trunk passed with JDK v1.8.0_74 +1 compile 6m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 3m 6s trunk passed +1 mvneclipse 1m 1s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 31s trunk passed +1 javadoc 1m 56s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 18s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 30s the patch passed +1 compile 6m 8s the patch passed with JDK v1.8.0_74 +1 javac 6m 8s the patch passed +1 compile 6m 50s the patch passed with JDK v1.7.0_95 +1 javac 6m 50s the patch passed -1 checkstyle 1m 4s root: patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 3m 27s the patch passed +1 mvneclipse 1m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 xml 0m 2s The patch has 1 ill-formed XML file(s). 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 22s the patch passed +1 javadoc 2m 9s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 38s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74. -1 unit 16m 3s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74. -1 unit 3m 47s hadoop-tools in the patch failed with JDK v1.8.0_74. +1 unit 0m 9s hadoop-tools-dist in the patch passed with JDK v1.8.0_74. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_95. -1 unit 16m 10s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 12s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. -1 unit 4m 14s hadoop-tools in the patch failed with JDK v1.7.0_95. +1 unit 0m 11s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 22s Patch generated 4 ASF License warnings. 103m 25s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.security.TestJNIGroupsMapping   hadoop.io.file.tfile.TestTFileSeqFileComparison   hadoop.fs.TestLocalFSFileContextCreateMkdir   hadoop.fs.shell.TestPathData   hadoop.io.compress.bzip2.TestBzip2CompressorDecompressor   hadoop.io.TestSequenceFileAppend   hadoop.fs.contract.localfs.TestLocalFSContractLoaded   hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays   hadoop.fs.shell.TestAclCommands   hadoop.ha.TestHealthMonitor   hadoop.ipc.TestMultipleProtocolServer   hadoop.ha.TestZKFailoverControllerStress   hadoop.io.serializer.TestSerializationFactory   hadoop.security.authorize.TestProxyServers   hadoop.fs.contract.ftp.TestFTPContractOpen   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem   hadoop.crypto.TestCryptoStreams   hadoop.util.TestGenericsUtil   hadoop.crypto.key.TestKeyProvider   hadoop.ipc.TestSocketFactory   hadoop.io.TestSecureIOUtils   hadoop.io.file.tfile.TestTFile   hadoop.fs.contract.ftp.TestFTPContractCreate   hadoop.http.TestHttpServerWebapps   hadoop.crypto.TestCryptoCodec   hadoop.fs.shell.TestCopyPreserveFlag   hadoop.fs.viewfs.TestFcMainOperationsLocalFs   hadoop.fs.TestLocalDirAllocator   hadoop.security.TestCompositeGroupMapping   hadoop.fs.contract.rawlocal.TestRawlocalContractSeek   hadoop.ipc.TestRPC   hadoop.fs.contract.localfs.TestLocalFSContractGetFileStatus   hadoop.http.TestGlobalFilter   hadoop.fs.shell.TestLs   hadoop.util.TestFindClass   hadoop.security.alias.TestCredentialProviderFactory   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   hadoop.http.TestHttpServerLifecycle   hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider   hadoop.net.TestStaticMapping   hadoop.fs.shell.find.TestName   hadoop.fs.TestFileContext   hadoop.ha.TestNodeFencer   hadoop.crypto.TestCryptoStreamsForLocalFS   hadoop.fs.TestDelegateToFileSystem   hadoop.net.TestScriptBasedMapping   hadoop.security.ssl.TestSSLFactory   hadoop.net.TestSwitchMapping   hadoop.security.TestHttpCrossOriginFilterInitializer   hadoop.conf.TestConfigurationSubclass   hadoop.fs.TestTruncatedInputBug   hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir   hadoop.http.TestAuthenticationSessionCookie   hadoop.fs.TestGetFileBlockLocations   hadoop.fs.viewfs.TestFcPermissionsLocalFs   hadoop.security.TestDoAsEffectiveUser   hadoop.ipc.TestDecayRpcScheduler   hadoop.fs.TestLocalFSFileContextMainOperations   hadoop.fs.viewfs.TestViewFsURIs   hadoop.crypto.key.TestKeyProviderDelegationTokenExtension   hadoop.ipc.TestMiniRPCBenchmark   hadoop.fs.TestStat   hadoop.security.TestLdapGroupsMapping   hadoop.io.TestSequenceFileSerialization   hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays   hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.http.TestHttpCookieFlag   hadoop.log.TestLogLevel   hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem   hadoop.http.TestPathFilter   hadoop.ipc.TestSaslRPC   hadoop.io.TestMapFile   hadoop.util.TestNativeCrc32   hadoop.io.file.tfile.TestVLong   hadoop.service.TestServiceLifecycle   hadoop.fs.TestLocalFileSystemPermission   hadoop.fs.TestPath   hadoop.fs.viewfs.TestChRootedFs   hadoop.crypto.random.TestOsSecureRandom   hadoop.util.TestGenericOptionsParser   hadoop.io.file.tfile.TestTFileStreams   hadoop.fs.contract.localfs.TestLocalFSContractMkdir   hadoop.fs.contract.localfs.TestLocalFSContractRename   hadoop.fs.TestFcLocalFsPermission   hadoop.ha.TestSshFenceByTcpPort   hadoop.fs.TestFileUtil   hadoop.io.TestSequenceFile   hadoop.ipc.TestWeightedRoundRobinMultiplexer   hadoop.io.serializer.avro.TestAvroSerialization   hadoop.ha.TestShellCommandFencer   hadoop.security.authorize.TestServiceAuthorization   hadoop.fs.permission.TestFsPermission   hadoop.fs.TestS3_LocalFileContextURI   hadoop.crypto.TestCryptoStreamsWithJceAesCtrCryptoCodec   hadoop.fs.viewfs.TestViewfsFileStatus   hadoop.fs.contract.rawlocal.TestRawlocalContractRename   hadoop.io.compress.TestCodecPool   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.fs.contract.localfs.TestLocalFSContractSeek   hadoop.security.TestShellBasedIdMapping   hadoop.ipc.TestProtoBufRpc   hadoop.http.TestHttpServerLogs   hadoop.fs.shell.TestXAttrCommands   hadoop.fs.contract.rawlocal.TestRawlocalContractOpen   hadoop.ipc.TestRPCServerShutdown   hadoop.fs.contract.localfs.TestLocalFSContractDelete   hadoop.io.TestWritable   hadoop.security.TestKDiagNoKDC   hadoop.conf.TestReconfiguration   hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs   hadoop.cli.TestCLI   hadoop.security.token.delegation.web.TestWebDelegationToken   hadoop.fs.shell.TestCount   hadoop.fs.shell.find.TestIname   hadoop.io.erasurecode.coder.TestHHXORErasureCoder   hadoop.http.TestServletFilter   hadoop.ipc.TestProtoBufRPCCompatibility   hadoop.fs.TestFilterFileSystem   hadoop.fs.TestFsShellReturnCode   hadoop.fs.viewfs.TestViewFsConfig   hadoop.fs.TestFcLocalFsUtil   hadoop.io.file.tfile.TestTFileNoneCodecsStreams   hadoop.conf.TestConfServlet   hadoop.ha.TestZKFailoverController   hadoop.metrics2.source.TestJvmMetrics   hadoop.io.file.tfile.TestTFileSplit   hadoop.io.erasurecode.coder.TestRSErasureCoder   hadoop.ipc.TestIPC   hadoop.security.TestGroupFallback   hadoop.fs.shell.find.TestPrint0   hadoop.io.TestBloomMapFile   hadoop.net.TestNetUtils   hadoop.crypto.TestCryptoStreamsNormal   hadoop.ipc.TestIdentityProviders   hadoop.io.TestGenericWritable   hadoop.fs.TestFsShellCopy   hadoop.io.file.tfile.TestTFileComparator2   hadoop.fs.TestFileSystemCanonicalization   hadoop.fs.shell.TestCopy   hadoop.fs.contract.rawlocal.TestRawlocalContractCreate   hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks   hadoop.ipc.TestServer   hadoop.http.TestSSLHttpServer   hadoop.fs.contract.rawlocal.TestRawlocalContractDelete   hadoop.fs.contract.localfs.TestLocalFSContractOpen   hadoop.io.file.tfile.TestTFileSeek   hadoop.security.TestKDiag   hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport   hadoop.io.compress.zlib.TestZlibCompressorDecompressor   hadoop.fs.TestChecksumFileSystem   hadoop.ipc.TestRPCWaitForProxy   hadoop.security.token.delegation.web.TestDelegationTokenManager   hadoop.fs.shell.find.TestFind   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec   hadoop.io.TestSequenceFileSync   hadoop.fs.shell.find.TestPrint   hadoop.fs.contract.ftp.TestFTPContractMkdir   hadoop.ha.TestFailoverController   hadoop.fs.viewfs.TestChRootedFileSystem   hadoop.security.authorize.TestProxyUsers   hadoop.security.alias.TestCredShell   hadoop.fs.shell.TestMove   hadoop.io.compress.TestCompressorDecompressor   hadoop.metrics2.sink.TestRollingFileSystemSink   hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays   hadoop.fs.TestSymlinkLocalFSFileSystem   hadoop.ipc.TestIPCServerResponder   hadoop.util.TestNativeLibraryChecker   hadoop.security.TestLdapGroupsMappingWithPosixGroup   hadoop.fs.TestFSMainOperationsLocalFileSystem   hadoop.fs.contract.localfs.TestLocalFSContractAppend   hadoop.security.TestUGILoginFromKeytab   hadoop.fs.contract.localfs.TestLocalFSContractSetTimes   hadoop.security.TestSecurityUtil   hadoop.net.TestTableMapping   hadoop.fs.TestLocalFsFCStatistics   hadoop.fs.TestFileSystemInitialization   hadoop.fs.contract.rawlocal.TestRawlocalContractSetTimes   hadoop.util.hash.TestHash   hadoop.io.TestDefaultStringifier   hadoop.fs.contract.ftp.TestFTPContractDelete   hadoop.io.serializer.TestWritableSerialization   hadoop.net.TestScriptBasedMappingWithDependency   hadoop.io.TestArrayFile   hadoop.io.compress.TestCodecFactory   hadoop.security.TestWhitelistBasedResolver   hadoop.fs.TestFsShell   hadoop.fs.TestHarFileSystemBasics   hadoop.ipc.TestFairCallQueue   hadoop.security.authorize.TestAccessControlList   hadoop.fs.contract.ftp.TestFTPContractRename   hadoop.fs.TestFileContextResolveAfs   hadoop.io.file.tfile.TestTFileComparators   hadoop.fs.TestListFiles   hadoop.security.TestProxyUserFromEnv   hadoop.ipc.TestRPCCallBenchmark   hadoop.fs.shell.TestTextCommand   hadoop.io.file.tfile.TestTFileLzoCodecsStreams   hadoop.fs.TestHarFileSystem   hadoop.ha.TestHAAdmin   hadoop.fs.contract.localfs.TestLocalFSContractCreate   hadoop.io.retry.TestDefaultRetryPolicy   hadoop.ipc.TestRPCCompatibility   hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem   hadoop.conf.TestGetInstances   hadoop.fs.viewfs.TestViewFileSystemDelegation   hadoop.fs.viewfs.TestViewFsTrash   hadoop.io.file.tfile.TestTFileUnsortedByteArrays   hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress   hadoop.io.compress.TestCodec   hadoop.security.TestGroupsCaching   hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus   hadoop.fs.TestLocalFileSystem   hadoop.tracing.TestTraceUtils   hadoop.fs.TestLocal_S3FileContextURI   hadoop.crypto.key.TestKeyShell   hadoop.security.TestAuthenticationFilter   hadoop.util.TestDiskChecker   hadoop.conf.TestDeprecatedKeys   hadoop.fs.viewfs.TestViewFsLocalFs   hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays   hadoop.fs.sftp.TestSFTPFileSystem   hadoop.security.TestUserFromEnv   hadoop.crypto.key.TestKeyProviderFactory   hadoop.io.compress.TestCompressionStreamReuse   hadoop.conf.TestConfigurationDeprecation   hadoop.io.TestSetFile   hadoop.crypto.key.TestKeyProviderCryptoExtension   hadoop.conf.TestConfiguration   hadoop.util.TestNodeHealthScriptRunner   hadoop.fs.viewfs.TestFcCreateMkdirLocalFs   hadoop.fs.TestFileContextDeleteOnExit   hadoop.io.file.tfile.TestTFileByteArrays   hadoop.jmx.TestJMXJsonServlet   hadoop.http.lib.TestStaticUserWebFilter   hadoop.http.TestHttpServer   hadoop.fs.TestTrash   hadoop.fs.TestDU   hadoop.fs.TestAvroFSInput   hadoop.security.TestUserGroupInformation   hadoop.tools.TestHadoopArchives   hadoop.tools.TestDistCpSync   hadoop.tools.mapred.TestCopyCommitter   hadoop.tools.TestDistCpWithXAttrs   hadoop.tools.TestDistCpWithRawXAttrs   hadoop.tools.TestIntegration   hadoop.tools.mapred.TestCopyOutputFormat   hadoop.tools.mapred.TestUniformSizeInputFormat   hadoop.tools.TestOptionsParser   hadoop.tools.TestGlobbedCopyListing   hadoop.tools.mapred.lib.TestDynamicInputFormat   hadoop.tools.mapred.TestCopyMapper   hadoop.tools.TestDistCpSystem   hadoop.tools.TestDistCpViewFs   hadoop.tools.TestFileBasedCopyListing   hadoop.tools.TestCopyListing   hadoop.tools.TestExternalCall   hadoop.tools.util.TestDistCpUtils   hadoop.tools.mapred.TestRetriableFileCopyCommand   hadoop.tools.TestDistCpWithAcls   hadoop.mapred.gridmix.TestFileQueue   hadoop.mapred.gridmix.TestGridMixClasses   hadoop.mapred.gridmix.TestHighRamJob   hadoop.mapred.gridmix.TestPseudoLocalFs   hadoop.mapred.gridmix.TestDistCacheEmulation   hadoop.mapred.gridmix.TestSleepJob   hadoop.mapred.gridmix.TestUserResolve   hadoop.mapred.gridmix.TestCompressionEmulationUtils   hadoop.mapred.gridmix.TestFilePool   hadoop.mapred.gridmix.TestRecordFactory   hadoop.mapred.gridmix.TestGridmixSummary   hadoop.mapred.gridmix.TestLoadJob   hadoop.mapred.gridmix.TestGridmixSubmission   hadoop.mapred.gridmix.TestResourceUsageEmulators   hadoop.mapred.gridmix.TestGridmixMemoryEmulation   hadoop.tools.TestDistCh   hadoop.fs.azure.TestNativeAzureFileSystemMocked   hadoop.fs.azure.TestBlobMetadata   hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater   hadoop.fs.azure.contract.TestAzureNativeContractMkdir   hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations   hadoop.fs.azure.TestOutOfBandAzureBlobOperations   hadoop.fs.azure.contract.TestAzureNativeContractRename   hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck   hadoop.fs.azure.TestNativeAzureFileSystemLive   hadoop.fs.azure.TestFileSystemOperationExceptionHandling   hadoop.fs.azure.TestNativeAzureFSPageBlobLive   hadoop.fs.azure.TestWasbUriAndConfiguration   hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded   hadoop.fs.azure.TestWasbFsck   hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive   hadoop.fs.azure.TestNativeAzureFileSystemContractLive   hadoop.fs.azure.TestNativeAzureFileSystemConcurrency   hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem   hadoop.fs.azure.contract.TestAzureNativeContractDelete   hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked   hadoop.fs.azure.TestAzureConcurrentOutOfBandIo   hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation   hadoop.fs.azure.TestNativeAzureFileSystemAppend   hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive   hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList   hadoop.fs.azure.contract.TestAzureNativeContractOpen   hadoop.fs.azure.TestBlobDataValidation   hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite   hadoop.fs.azure.contract.TestAzureNativeContractAppend   hadoop.fs.azure.contract.TestAzureNativeContractSeek   hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus   hadoop.fs.azure.TestBlobTypeSpeedDifference   hadoop.fs.azure.TestContainerChecks   hadoop.fs.azure.TestAzureFileSystemErrorConditions   hadoop.fs.azure.TestNativeAzureFileSystemClientLogging   hadoop.fs.azure.TestNativeAzureFileSystemContractMocked   hadoop.fs.azure.contract.TestAzureNativeContractCreate   hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator   hadoop.streaming.TestStreamReduceNone   hadoop.streaming.TestStreamingBackground   hadoop.streaming.TestStreamXmlRecordReader   hadoop.streaming.TestUnconsumedInput   hadoop.streaming.TestStreamingCounters   hadoop.streaming.TestStreamingOutputOnlyKeys   hadoop.streaming.TestTypedBytesStreaming   hadoop.streaming.TestStreamingExitStatus   hadoop.streaming.TestDumpTypedBytes   hadoop.streaming.io.TestKeyOnlyTextOutputReader   hadoop.streaming.TestStreamAggregate   hadoop.streaming.TestMRFramework   hadoop.streaming.TestMultipleArchiveFiles   hadoop.streaming.TestStreaming   hadoop.streaming.TestStreamingKeyValue   hadoop.streaming.TestStreamJob   hadoop.streaming.TestStreamingFailure   hadoop.streaming.TestStreamingCombiner   hadoop.streaming.TestStreamingStderr   hadoop.streaming.TestStreamDataProtocol   hadoop.streaming.TestStreamXmlMultipleRecords   hadoop.streaming.TestAutoInputFormat   hadoop.streaming.mapreduce.TestStreamXmlRecordReader   hadoop.streaming.TestStreamingBadRecords   hadoop.streaming.TestStreamingSeparator   hadoop.streaming.TestSymLink   hadoop.streaming.TestStreamingOutputKeyValueTypes   hadoop.streaming.TestMultipleCachefiles   hadoop.streaming.TestGzipInput   hadoop.streaming.TestLoadTypedBytes   hadoop.streaming.TestRawBytesStreaming   hadoop.streaming.TestFileArgs   hadoop.contrib.utils.join.TestDataJoin   hadoop.yarn.sls.appmaster.TestAMSimulator   hadoop.yarn.sls.nodemanager.TestNMSimulator   hadoop.yarn.sls.TestSLSRunner JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.fs.TestFileSystemCaching   org.apache.hadoop.io.nativeio.TestNativeIO JDK v1.7.0_95 Failed junit tests hadoop.security.TestJNIGroupsMapping   hadoop.io.file.tfile.TestTFileSeqFileComparison   hadoop.fs.TestLocalFSFileContextCreateMkdir   hadoop.fs.shell.TestPathData   hadoop.io.compress.bzip2.TestBzip2CompressorDecompressor   hadoop.io.TestSequenceFileAppend   hadoop.fs.contract.localfs.TestLocalFSContractLoaded   hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays   hadoop.fs.shell.TestAclCommands   hadoop.ha.TestHealthMonitor   hadoop.ipc.TestMultipleProtocolServer   hadoop.ha.TestZKFailoverControllerStress   hadoop.io.serializer.TestSerializationFactory   hadoop.security.authorize.TestProxyServers   hadoop.fs.contract.ftp.TestFTPContractOpen   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem   hadoop.crypto.TestCryptoStreams   hadoop.util.TestGenericsUtil   hadoop.crypto.key.TestKeyProvider   hadoop.ipc.TestSocketFactory   hadoop.io.TestSecureIOUtils   hadoop.io.file.tfile.TestTFile   hadoop.fs.contract.ftp.TestFTPContractCreate   hadoop.http.TestHttpServerWebapps   hadoop.crypto.TestCryptoCodec   hadoop.fs.shell.TestCopyPreserveFlag   hadoop.fs.viewfs.TestFcMainOperationsLocalFs   hadoop.fs.TestLocalDirAllocator   hadoop.security.TestCompositeGroupMapping   hadoop.fs.contract.rawlocal.TestRawlocalContractSeek   hadoop.ipc.TestRPC   hadoop.fs.contract.localfs.TestLocalFSContractGetFileStatus   hadoop.http.TestGlobalFilter   hadoop.fs.shell.TestLs   hadoop.util.TestFindClass   hadoop.security.alias.TestCredentialProviderFactory   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   hadoop.http.TestHttpServerLifecycle   hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider   hadoop.net.TestStaticMapping   hadoop.fs.shell.find.TestName   hadoop.fs.TestFileContext   hadoop.ha.TestNodeFencer   hadoop.crypto.TestCryptoStreamsForLocalFS   hadoop.fs.TestDelegateToFileSystem   hadoop.net.TestScriptBasedMapping   hadoop.security.ssl.TestSSLFactory   hadoop.net.TestSwitchMapping   hadoop.security.TestHttpCrossOriginFilterInitializer   hadoop.conf.TestConfigurationSubclass   hadoop.fs.TestTruncatedInputBug   hadoop.fs.contract.rawlocal.TestRawlocalContractMkdir   hadoop.http.TestAuthenticationSessionCookie   hadoop.fs.TestGetFileBlockLocations   hadoop.fs.viewfs.TestFcPermissionsLocalFs   hadoop.security.TestDoAsEffectiveUser   hadoop.ipc.TestDecayRpcScheduler   hadoop.fs.TestLocalFSFileContextMainOperations   hadoop.fs.viewfs.TestViewFsURIs   hadoop.crypto.key.TestKeyProviderDelegationTokenExtension   hadoop.ipc.TestMiniRPCBenchmark   hadoop.fs.TestStat   hadoop.security.TestLdapGroupsMapping   hadoop.io.TestSequenceFileSerialization   hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays   hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.http.TestHttpCookieFlag   hadoop.log.TestLogLevel   hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem   hadoop.http.TestPathFilter   hadoop.ipc.TestSaslRPC   hadoop.io.TestMapFile   hadoop.util.TestNativeCrc32   hadoop.io.file.tfile.TestVLong   hadoop.service.TestServiceLifecycle   hadoop.fs.TestLocalFileSystemPermission   hadoop.fs.TestPath   hadoop.fs.viewfs.TestChRootedFs   hadoop.crypto.random.TestOsSecureRandom   hadoop.util.TestGenericOptionsParser   hadoop.io.file.tfile.TestTFileStreams   hadoop.fs.contract.localfs.TestLocalFSContractMkdir   hadoop.fs.contract.localfs.TestLocalFSContractRename   hadoop.fs.TestFcLocalFsPermission   hadoop.ha.TestSshFenceByTcpPort   hadoop.fs.TestFileUtil   hadoop.io.TestSequenceFile   hadoop.ipc.TestWeightedRoundRobinMultiplexer   hadoop.io.serializer.avro.TestAvroSerialization   hadoop.ha.TestShellCommandFencer   hadoop.security.authorize.TestServiceAuthorization   hadoop.fs.permission.TestFsPermission   hadoop.fs.TestS3_LocalFileContextURI   hadoop.crypto.TestCryptoStreamsWithJceAesCtrCryptoCodec   hadoop.fs.viewfs.TestViewfsFileStatus   hadoop.fs.contract.rawlocal.TestRawlocalContractRename   hadoop.io.compress.TestCodecPool   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.fs.contract.localfs.TestLocalFSContractSeek   hadoop.security.TestShellBasedIdMapping   hadoop.ipc.TestProtoBufRpc   hadoop.http.TestHttpServerLogs   hadoop.fs.shell.TestXAttrCommands   hadoop.fs.contract.rawlocal.TestRawlocalContractOpen   hadoop.ipc.TestRPCServerShutdown   hadoop.fs.contract.localfs.TestLocalFSContractDelete   hadoop.io.TestWritable   hadoop.security.TestKDiagNoKDC   hadoop.conf.TestReconfiguration   hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs   hadoop.cli.TestCLI   hadoop.security.token.delegation.web.TestWebDelegationToken   hadoop.fs.shell.TestCount   hadoop.fs.shell.find.TestIname   hadoop.io.erasurecode.coder.TestHHXORErasureCoder   hadoop.http.TestServletFilter   hadoop.ipc.TestProtoBufRPCCompatibility   hadoop.fs.TestFilterFileSystem   hadoop.fs.TestFsShellReturnCode   hadoop.fs.viewfs.TestViewFsConfig   hadoop.fs.TestFcLocalFsUtil   hadoop.io.file.tfile.TestTFileNoneCodecsStreams   hadoop.conf.TestConfServlet   hadoop.ha.TestZKFailoverController   hadoop.metrics2.source.TestJvmMetrics   hadoop.io.file.tfile.TestTFileSplit   hadoop.io.erasurecode.coder.TestRSErasureCoder   hadoop.ipc.TestIPC   hadoop.security.TestGroupFallback   hadoop.fs.shell.find.TestPrint0   hadoop.io.TestBloomMapFile   hadoop.net.TestNetUtils   hadoop.crypto.TestCryptoStreamsNormal   hadoop.ipc.TestIdentityProviders   hadoop.io.TestGenericWritable   hadoop.fs.TestFsShellCopy   hadoop.io.file.tfile.TestTFileComparator2   hadoop.fs.TestFileSystemCanonicalization   hadoop.fs.shell.TestCopy   hadoop.fs.contract.rawlocal.TestRawlocalContractCreate   hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks   hadoop.ipc.TestServer   hadoop.http.TestSSLHttpServer   hadoop.fs.contract.rawlocal.TestRawlocalContractDelete   hadoop.fs.contract.localfs.TestLocalFSContractOpen   hadoop.io.file.tfile.TestTFileSeek   hadoop.security.TestKDiag   hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport   hadoop.io.compress.zlib.TestZlibCompressorDecompressor   hadoop.fs.TestChecksumFileSystem   hadoop.ipc.TestRPCWaitForProxy   hadoop.security.token.delegation.web.TestDelegationTokenManager   hadoop.fs.shell.find.TestFind   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec   hadoop.io.TestSequenceFileSync   hadoop.fs.shell.find.TestPrint   hadoop.fs.contract.ftp.TestFTPContractMkdir   hadoop.ha.TestFailoverController   hadoop.fs.viewfs.TestChRootedFileSystem   hadoop.security.authorize.TestProxyUsers   hadoop.security.alias.TestCredShell   hadoop.fs.shell.TestMove   hadoop.io.compress.TestCompressorDecompressor   hadoop.metrics2.sink.TestRollingFileSystemSink   hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays   hadoop.fs.TestSymlinkLocalFSFileSystem   hadoop.ipc.TestIPCServerResponder   hadoop.util.TestNativeLibraryChecker   hadoop.security.TestLdapGroupsMappingWithPosixGroup   hadoop.fs.TestFSMainOperationsLocalFileSystem   hadoop.fs.contract.localfs.TestLocalFSContractAppend   hadoop.security.TestUGILoginFromKeytab   hadoop.fs.contract.localfs.TestLocalFSContractSetTimes   hadoop.security.TestSecurityUtil   hadoop.net.TestTableMapping   hadoop.fs.TestLocalFsFCStatistics   hadoop.fs.TestFileSystemInitialization   hadoop.fs.contract.rawlocal.TestRawlocalContractSetTimes   hadoop.util.hash.TestHash   hadoop.io.TestDefaultStringifier   hadoop.fs.contract.ftp.TestFTPContractDelete   hadoop.io.serializer.TestWritableSerialization   hadoop.net.TestScriptBasedMappingWithDependency   hadoop.io.TestArrayFile   hadoop.io.compress.TestCodecFactory   hadoop.security.TestWhitelistBasedResolver   hadoop.fs.TestFsShell   hadoop.fs.TestHarFileSystemBasics   hadoop.ipc.TestFairCallQueue   hadoop.security.authorize.TestAccessControlList   hadoop.fs.contract.ftp.TestFTPContractRename   hadoop.fs.TestFileContextResolveAfs   hadoop.io.file.tfile.TestTFileComparators   hadoop.fs.TestListFiles   hadoop.security.TestProxyUserFromEnv   hadoop.ipc.TestRPCCallBenchmark   hadoop.fs.shell.TestTextCommand   hadoop.io.file.tfile.TestTFileLzoCodecsStreams   hadoop.fs.TestHarFileSystem   hadoop.ha.TestHAAdmin   hadoop.fs.contract.localfs.TestLocalFSContractCreate   hadoop.io.retry.TestDefaultRetryPolicy   hadoop.ipc.TestRPCCompatibility   hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem   hadoop.conf.TestGetInstances   hadoop.fs.viewfs.TestViewFileSystemDelegation   hadoop.fs.viewfs.TestViewFsTrash   hadoop.io.file.tfile.TestTFileUnsortedByteArrays   hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress   hadoop.io.compress.TestCodec   hadoop.security.TestGroupsCaching   hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus   hadoop.fs.TestLocalFileSystem   hadoop.tracing.TestTraceUtils   hadoop.fs.TestLocal_S3FileContextURI   hadoop.crypto.key.TestKeyShell   hadoop.security.TestAuthenticationFilter   hadoop.util.TestDiskChecker   hadoop.conf.TestDeprecatedKeys   hadoop.fs.viewfs.TestViewFsLocalFs   hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays   hadoop.fs.sftp.TestSFTPFileSystem   hadoop.security.TestUserFromEnv   hadoop.crypto.key.TestKeyProviderFactory   hadoop.io.compress.TestCompressionStreamReuse   hadoop.conf.TestConfigurationDeprecation   hadoop.io.TestSetFile   hadoop.crypto.key.TestKeyProviderCryptoExtension   hadoop.conf.TestConfiguration   hadoop.util.TestNodeHealthScriptRunner   hadoop.fs.viewfs.TestFcCreateMkdirLocalFs   hadoop.fs.TestFileContextDeleteOnExit   hadoop.io.file.tfile.TestTFileByteArrays   hadoop.jmx.TestJMXJsonServlet   hadoop.http.lib.TestStaticUserWebFilter   hadoop.http.TestHttpServer   hadoop.fs.TestTrash   hadoop.fs.TestDU   hadoop.fs.TestAvroFSInput   hadoop.security.TestUserGroupInformation   hadoop.tools.TestHadoopArchives   hadoop.tools.TestDistCpSync   hadoop.tools.mapred.TestCopyCommitter   hadoop.tools.TestDistCpWithXAttrs   hadoop.tools.TestDistCpWithRawXAttrs   hadoop.tools.TestIntegration   hadoop.tools.mapred.TestCopyOutputFormat   hadoop.tools.mapred.TestUniformSizeInputFormat   hadoop.tools.TestOptionsParser   hadoop.tools.TestGlobbedCopyListing   hadoop.tools.mapred.lib.TestDynamicInputFormat   hadoop.tools.mapred.TestCopyMapper   hadoop.tools.TestDistCpSystem   hadoop.tools.TestDistCpViewFs   hadoop.tools.TestFileBasedCopyListing   hadoop.tools.TestCopyListing   hadoop.tools.TestExternalCall   hadoop.tools.util.TestDistCpUtils   hadoop.tools.mapred.TestRetriableFileCopyCommand   hadoop.tools.TestDistCpWithAcls   hadoop.mapred.gridmix.TestFileQueue   hadoop.mapred.gridmix.TestGridMixClasses   hadoop.mapred.gridmix.TestHighRamJob   hadoop.mapred.gridmix.TestPseudoLocalFs   hadoop.mapred.gridmix.TestDistCacheEmulation   hadoop.mapred.gridmix.TestSleepJob   hadoop.mapred.gridmix.TestUserResolve   hadoop.mapred.gridmix.TestCompressionEmulationUtils   hadoop.mapred.gridmix.TestFilePool   hadoop.mapred.gridmix.TestRecordFactory   hadoop.mapred.gridmix.TestGridmixSummary   hadoop.mapred.gridmix.TestLoadJob   hadoop.mapred.gridmix.TestGridmixSubmission   hadoop.mapred.gridmix.TestResourceUsageEmulators   hadoop.mapred.gridmix.TestGridmixMemoryEmulation   hadoop.tools.TestDistCh   hadoop.fs.azure.TestNativeAzureFileSystemMocked   hadoop.fs.azure.TestBlobMetadata   hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater   hadoop.fs.azure.contract.TestAzureNativeContractMkdir   hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations   hadoop.fs.azure.TestOutOfBandAzureBlobOperations   hadoop.fs.azure.contract.TestAzureNativeContractRename   hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck   hadoop.fs.azure.TestNativeAzureFileSystemLive   hadoop.fs.azure.TestFileSystemOperationExceptionHandling   hadoop.fs.azure.TestNativeAzureFSPageBlobLive   hadoop.fs.azure.TestWasbUriAndConfiguration   hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded   hadoop.fs.azure.TestWasbFsck   hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive   hadoop.fs.azure.TestNativeAzureFileSystemContractLive   hadoop.fs.azure.TestNativeAzureFileSystemConcurrency   hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem   hadoop.fs.azure.contract.TestAzureNativeContractDelete   hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked   hadoop.fs.azure.TestAzureConcurrentOutOfBandIo   hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation   hadoop.fs.azure.TestNativeAzureFileSystemAppend   hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive   hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList   hadoop.fs.azure.contract.TestAzureNativeContractOpen   hadoop.fs.azure.TestBlobDataValidation   hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite   hadoop.fs.azure.contract.TestAzureNativeContractAppend   hadoop.fs.azure.contract.TestAzureNativeContractSeek   hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus   hadoop.fs.azure.TestBlobTypeSpeedDifference   hadoop.fs.azure.TestContainerChecks   hadoop.fs.azure.TestAzureFileSystemErrorConditions   hadoop.fs.azure.TestNativeAzureFileSystemClientLogging   hadoop.fs.azure.TestNativeAzureFileSystemContractMocked   hadoop.fs.azure.contract.TestAzureNativeContractCreate   hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator   hadoop.streaming.TestStreamReduceNone   hadoop.streaming.TestStreamingBackground   hadoop.streaming.TestStreamXmlRecordReader   hadoop.streaming.TestUnconsumedInput   hadoop.streaming.TestStreamingCounters   hadoop.streaming.TestStreamingOutputOnlyKeys   hadoop.streaming.TestTypedBytesStreaming   hadoop.streaming.TestStreamingExitStatus   hadoop.streaming.TestDumpTypedBytes   hadoop.streaming.io.TestKeyOnlyTextOutputReader   hadoop.streaming.TestStreamAggregate   hadoop.streaming.TestMRFramework   hadoop.streaming.TestMultipleArchiveFiles   hadoop.streaming.TestStreaming   hadoop.streaming.TestStreamingKeyValue   hadoop.streaming.TestStreamJob   hadoop.streaming.TestStreamingFailure   hadoop.streaming.TestStreamingCombiner   hadoop.streaming.TestStreamingStderr   hadoop.streaming.TestStreamDataProtocol   hadoop.streaming.TestStreamXmlMultipleRecords   hadoop.streaming.TestAutoInputFormat   hadoop.streaming.mapreduce.TestStreamXmlRecordReader   hadoop.streaming.TestStreamingBadRecords   hadoop.streaming.TestStreamingSeparator   hadoop.streaming.TestSymLink   hadoop.streaming.TestStreamingOutputKeyValueTypes   hadoop.streaming.TestMultipleCachefiles   hadoop.streaming.TestGzipInput   hadoop.streaming.TestLoadTypedBytes   hadoop.streaming.TestRawBytesStreaming   hadoop.streaming.TestFileArgs   hadoop.contrib.utils.join.TestDataJoin   hadoop.yarn.sls.appmaster.TestAMSimulator   hadoop.yarn.sls.nodemanager.TestNMSimulator   hadoop.yarn.sls.TestSLSRunner JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.fs.TestFileSystemCaching   org.apache.hadoop.io.nativeio.TestNativeIO Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791749/HADOOP-12666-007.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ea081d554c13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fd1c09b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/diff-checkstyle-root.txt xml https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/xml.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8804/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Check-style issue fix for the comment missing ".".

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Check-style issue fix for the comment missing ".".
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 55s trunk passed with JDK v1.8.0_74
          +1 compile 7m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 3m 11s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 34s trunk passed
          +1 javadoc 2m 9s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 33s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 38s the patch passed
          +1 compile 6m 52s the patch passed with JDK v1.8.0_74
          +1 javac 6m 52s the patch passed
          +1 compile 7m 10s the patch passed with JDK v1.7.0_95
          +1 javac 7m 10s the patch passed
          +1 checkstyle 1m 7s the patch passed
          +1 mvnsite 3m 46s the patch passed
          +1 mvneclipse 1m 14s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 2m 40s the patch passed
          +1 javadoc 2m 23s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74.
          +1 unit 7m 30s hadoop-common in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 10s hadoop-tools-dist in the patch passed with JDK v1.8.0_74.
          +1 unit 33m 20s hadoop-tools in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_95.
          +1 unit 7m 37s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 15s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          +1 unit 32m 24s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          147m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791970/HADOOP-12666-008.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c796858ec2e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f86850b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist hadoop-tools U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 6m 55s trunk passed with JDK v1.8.0_74 +1 compile 7m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 3m 11s trunk passed +1 mvneclipse 0m 59s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 34s trunk passed +1 javadoc 2m 9s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 33s trunk passed with JDK v1.7.0_95 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 38s the patch passed +1 compile 6m 52s the patch passed with JDK v1.8.0_74 +1 javac 6m 52s the patch passed +1 compile 7m 10s the patch passed with JDK v1.7.0_95 +1 javac 7m 10s the patch passed +1 checkstyle 1m 7s the patch passed +1 mvnsite 3m 46s the patch passed +1 mvneclipse 1m 14s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 2m 40s the patch passed +1 javadoc 2m 23s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74. +1 unit 7m 30s hadoop-common in the patch passed with JDK v1.8.0_74. +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74. +1 unit 0m 10s hadoop-tools-dist in the patch passed with JDK v1.8.0_74. +1 unit 33m 20s hadoop-tools in the patch passed with JDK v1.8.0_74. +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_95. +1 unit 7m 37s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 15s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. +1 unit 32m 24s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 147m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791970/HADOOP-12666-008.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c796858ec2e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f86850b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist hadoop-tools U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8818/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          Hi Vishwajeet Dusane. Can you create a separate JIRA for the hadoop.hdfs.web patch (https://reviews.apache.org/r/44169/)? I think we should get that change reviewed and then we can have a Core Updates patch that doesn't mix up the packages.

          Show
          fabbri Aaron Fabbri added a comment - Hi Vishwajeet Dusane . Can you create a separate JIRA for the hadoop.hdfs.web patch ( https://reviews.apache.org/r/44169/)? I think we should get that change reviewed and then we can have a Core Updates patch that doesn't mix up the packages.
          Hide
          chris.douglas Chris Douglas added a comment -

          Can you create a separate JIRA for the hadoop.hdfs.web patch (https://reviews.apache.org/r/44169/)? I think we should get that change reviewed and then we can have a Core Updates patch that doesn't mix up the packages.

          Several (all?) reviewers have raised concerns about extending WebHDFS. The packaging issue is a minor symptom of the fragile coupling this creates, and it should be a goal to sever it. I'd like to avoid relaxing WebHDFS impl visibility to support this kind of extension, since 1) it's not sufficient to maintain WebHDFS as a public protocol and 2) we're unlikely to restrict visibility again, when this dependency is no longer required. Of the three approaches:

          1. Rely on package-private methods by adding classes to o.a.h.hdfs.web (current approach)
          2. Change WebHDFS to support (some) extensions (factories, protected callbacks, etc.)
          3. Copy/rename/shade the WebHDFS code this relies on

          The first seems least harmful, unless (2) can be implemented cleanly and without becoming too entangled in WebHDFS internals. Aaron Fabbri, Chris Nauroth, Sean Mackrory, Lei (Eddy) Xu, thoughts?

          Vishwajeet Dusane: Thanks for splitting the patch; this is significantly easier to review.

          The implementation of buffered I/O is complex. Some techniques, like ping-pong buffers, may not offer many advantages over allowing buffers to be GC'd. I haven't looked in detail, but the singleton BufferManager is not threadsafe and appears to be shared across all streams. Now that the patch is simpler, could you briefly summarize how the buffering works and why it wins over simpler, naive approaches?

          Show
          chris.douglas Chris Douglas added a comment - Can you create a separate JIRA for the hadoop.hdfs.web patch ( https://reviews.apache.org/r/44169/)? I think we should get that change reviewed and then we can have a Core Updates patch that doesn't mix up the packages. Several (all?) reviewers have raised concerns about extending WebHDFS. The packaging issue is a minor symptom of the fragile coupling this creates, and it should be a goal to sever it. I'd like to avoid relaxing WebHDFS impl visibility to support this kind of extension, since 1) it's not sufficient to maintain WebHDFS as a public protocol and 2) we're unlikely to restrict visibility again, when this dependency is no longer required. Of the three approaches: Rely on package-private methods by adding classes to o.a.h.hdfs.web (current approach) Change WebHDFS to support (some) extensions (factories, protected callbacks, etc.) Copy/rename/shade the WebHDFS code this relies on The first seems least harmful, unless (2) can be implemented cleanly and without becoming too entangled in WebHDFS internals. Aaron Fabbri , Chris Nauroth , Sean Mackrory , Lei (Eddy) Xu , thoughts? Vishwajeet Dusane : Thanks for splitting the patch; this is significantly easier to review. The implementation of buffered I/O is complex. Some techniques, like ping-pong buffers, may not offer many advantages over allowing buffers to be GC'd. I haven't looked in detail, but the singleton BufferManager is not threadsafe and appears to be shared across all streams. Now that the patch is simpler, could you briefly summarize how the buffering works and why it wins over simpler, naive approaches?
          Hide
          cnauroth Chris Nauroth added a comment -

          Chris Douglas, thank you for the excellent summary. I agree with the concerns about coupling with WebHDFS client internals. Asking for a WebHDFS patch was a way to make that coupling clearer to reviewers by showing relaxation of visibility explicitly.

          Something else to keep in mind is that the audience of contributors on WebHDFS is likely to be different from the audience of contributors on Azure Data Lake. If a change in the WebHDFS client forces a change in an Azure Data Lake subclass to satisfy compilation, then it's not guaranteed that the WebHDFS contributor is going to be well-equipped to make that change correctly and test it.

          The only way I see to break this coupling is for Azure Data Lake to implement its own client, without relying on inheritance from WebHDFS for code reuse. There are already significant protocol deviations from WebHDFS, so an ADL subclass is going to violate the Liskov substitution principle. (You can never substitute an instance of WebHdfsFileSystem with an instance of PrivateAzureDataLakeFileSystem, because that won't work correctly with the WebHDFS back-end.)

          If there is an eventual goal to move to a dedicated client codebase, then I can understand option 1 (reliance on package-private methods) as a short-term solution. I see your point about avoiding relaxing visibility of internals if it's only needed short-term.

          Show
          cnauroth Chris Nauroth added a comment - Chris Douglas , thank you for the excellent summary. I agree with the concerns about coupling with WebHDFS client internals. Asking for a WebHDFS patch was a way to make that coupling clearer to reviewers by showing relaxation of visibility explicitly. Something else to keep in mind is that the audience of contributors on WebHDFS is likely to be different from the audience of contributors on Azure Data Lake. If a change in the WebHDFS client forces a change in an Azure Data Lake subclass to satisfy compilation, then it's not guaranteed that the WebHDFS contributor is going to be well-equipped to make that change correctly and test it. The only way I see to break this coupling is for Azure Data Lake to implement its own client, without relying on inheritance from WebHDFS for code reuse. There are already significant protocol deviations from WebHDFS, so an ADL subclass is going to violate the Liskov substitution principle. (You can never substitute an instance of WebHdfsFileSystem with an instance of PrivateAzureDataLakeFileSystem , because that won't work correctly with the WebHDFS back-end.) If there is an eventual goal to move to a dedicated client codebase, then I can understand option 1 (reliance on package-private methods) as a short-term solution. I see your point about avoiding relaxing visibility of internals if it's only needed short-term.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Notes From Mar 9, 2016 Call w/ MSFT
          Who: Cloudera: Aaron Fabbri, Tony Wu, MSFT: Vishwajeet, Cathy, Chris Douglas, Shrikant
          Discussion
          1. Packaging / Code Structure

          • In general, ADL extension of WebHDFS would not be acceptable as long term solution
          • Webhdfs client not designed for extension.
          • Available options as of today
          • Option 1 vs 2 (refactor WebHDFS) vs 3 (copy paste code, bad)
          • Option 2 (MSFT): Need to make change to WebHDFS to accept ADL stuff. May be significant work.
          • Raise a separate JIRA for WebHDFS extension

          2. WebHDFS and ADL cannot co-exist problem if both follows OAuth2 authentication protocol

          • Near term: specify limitation of only one webhdfs client at a time w/ OAUTH. Ok to have Webhdfs non-oauth and ADL configured on same cluster. - AP: Vishwajeet to document as known limitation
          • Long term: v2 of adl connector that factors out webhdfs client commonality better

          3. Integrity / Semantics

          • Single writer semantics?
          • See leaseId in PrivateAzureDataLakeFileSystem::createNonRecursive()
          • Append semantics does not close connection hence the leaseId is not required.

          4. Action Items

          • [msft] Put webhdfs extension issue into a separate JIRA so folks from the community can comment. Do they prefer hadoop-azure-datalake mixes packages, or relaxing some method privacy, or suggest other approach? - Raised HDFS-9938
          • [msft] volatile not needed in addition to synchronized in BatchByteArrayInputStream - AP: Vishwajeet
          • [msft] Add to documentation: caveat for v1 where you can only have one WebHDFS (ADL or vanilla) with Oauth2 not both. - AP: Vishwajeet
          • [cloudera] Go over latest patches.
          • [cloudera] Reach out to other hadoop committers to see what else needs addressing before we can get committed.
          • [msft/cloudera] Start document on adl:// semantics, deltas versus HDFS, w/ and w/o FileStatusCache

          5. Follow Up Topics (homework / next meeting)

          • Follow up on append(). No leaseid. What is delta from HDFS semantics.
          • BufferManager purpose, coherency
          • For readahead, so multiple FSInputStreams can see the same buffer that was fetched with readahead.
          • Follow up on flushAsync() in write path (why / how)

          6. Future plan of ADL client implementation

          • Share with community about future plans
          • Versioning
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Notes From Mar 9, 2016 Call w/ MSFT Who: Cloudera: Aaron Fabbri, Tony Wu, MSFT: Vishwajeet, Cathy, Chris Douglas, Shrikant Discussion 1. Packaging / Code Structure In general, ADL extension of WebHDFS would not be acceptable as long term solution Webhdfs client not designed for extension. Available options as of today Option 1 vs 2 (refactor WebHDFS) vs 3 (copy paste code, bad) Option 2 (MSFT): Need to make change to WebHDFS to accept ADL stuff. May be significant work. Raise a separate JIRA for WebHDFS extension 2. WebHDFS and ADL cannot co-exist problem if both follows OAuth2 authentication protocol Near term: specify limitation of only one webhdfs client at a time w/ OAUTH. Ok to have Webhdfs non-oauth and ADL configured on same cluster. - AP: Vishwajeet to document as known limitation Long term: v2 of adl connector that factors out webhdfs client commonality better 3. Integrity / Semantics Single writer semantics? See leaseId in PrivateAzureDataLakeFileSystem::createNonRecursive() Append semantics does not close connection hence the leaseId is not required. 4. Action Items [msft] Put webhdfs extension issue into a separate JIRA so folks from the community can comment. Do they prefer hadoop-azure-datalake mixes packages, or relaxing some method privacy, or suggest other approach? - Raised HDFS-9938 [msft] volatile not needed in addition to synchronized in BatchByteArrayInputStream - AP: Vishwajeet [msft] Add to documentation: caveat for v1 where you can only have one WebHDFS (ADL or vanilla) with Oauth2 not both. - AP: Vishwajeet [cloudera] Go over latest patches. [cloudera] Reach out to other hadoop committers to see what else needs addressing before we can get committed. [msft/cloudera] Start document on adl:// semantics, deltas versus HDFS, w/ and w/o FileStatusCache 5. Follow Up Topics (homework / next meeting) Follow up on append(). No leaseid. What is delta from HDFS semantics. BufferManager purpose, coherency For readahead, so multiple FSInputStreams can see the same buffer that was fetched with readahead. Follow up on flushAsync() in write path (why / how) 6. Future plan of ADL client implementation Share with community about future plans Versioning
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Nauroth Thanks a lot Chris. We are working on the feasibility of having dedicated client and timeline for that. Apart from Create semantics and no redirects operation during Create, Append and Read, do you think ADL is deviating on any other protocol as well? If yes then could you please highlight them so we can take it up mostly as separate JIRA? Our intention is to be as close to Hadoop semantics as possible.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Nauroth Thanks a lot Chris. We are working on the feasibility of having dedicated client and timeline for that. Apart from Create semantics and no redirects operation during Create, Append and Read, do you think ADL is deviating on any other protocol as well? If yes then could you please highlight them so we can take it up mostly as separate JIRA? Our intention is to be as close to Hadoop semantics as possible.
          Hide
          cnauroth Chris Nauroth added a comment -

          Notes From Mar 9, 2016 Call w/ MSFT

          I really should have been in this meeting. More importantly, any out-of-band meeting like this should be announced prior for full disclosure with the Apache community. Was there an announcement that I just missed in the deluge of email? If not, then please pre-announce any similar meetings in the future.

          Thank you for posting the summary publicly here though. That's exactly the right thing to do.

          Show
          cnauroth Chris Nauroth added a comment - Notes From Mar 9, 2016 Call w/ MSFT I really should have been in this meeting. More importantly, any out-of-band meeting like this should be announced prior for full disclosure with the Apache community. Was there an announcement that I just missed in the deluge of email? If not, then please pre-announce any similar meetings in the future. Thank you for posting the summary publicly here though. That's exactly the right thing to do.
          Hide
          cnauroth Chris Nauroth added a comment -

          The create/append/flush sequence is hugely different behavior. At the protocol layer, there is the addition of the flush parameter, which is a deviation from stock WebHDFS. Basically any of the custom *Param classes represent deviations from WebHDFS protocol: leaseId, ADLFeatureSet, etc.

          At the client layer, the aggressive client-side caching and buffering in the name of performance creates different behavior from stock WebHDFS. I and others have called out that while perhaps you don't observe anything to be broken right now, that's no guarantee that cache consistency won't become a problem for certain applications. This is not a wire protocol difference, but it is a significant deviation in behavior from stock WebHDFS.

          At this point, it appears that the ADL protocol, while heavily inspired by the WebHDFS protocol, is not really a compatible match. It is its own protocol with its own unique requirements for clients to use it correctly and use it well. Accidentally connecting the ADL client to an HDFS cluster would be disastrous. The create/append/flush sequence would cause massive unsustainable load to the NameNode in terms of RPC calls and edit logging. Client write latency would be unacceptable. Likewise, accidentally connecting the stock WebHDFS client to ADL seems to yield unacceptable performance for ADL.

          It is these large deviations that lead me to conclude the best choice is a dedicated client distinct from the WebHDFS client code. Having full control of that client gives us the opportunity to provide the best possible user experience with ADL. As I've stated before though, I can accept a short-term plan of some code reuse with the WebHDFS client.

          Show
          cnauroth Chris Nauroth added a comment - The create/append/flush sequence is hugely different behavior. At the protocol layer, there is the addition of the flush parameter, which is a deviation from stock WebHDFS. Basically any of the custom *Param classes represent deviations from WebHDFS protocol: leaseId, ADLFeatureSet, etc. At the client layer, the aggressive client-side caching and buffering in the name of performance creates different behavior from stock WebHDFS. I and others have called out that while perhaps you don't observe anything to be broken right now, that's no guarantee that cache consistency won't become a problem for certain applications. This is not a wire protocol difference, but it is a significant deviation in behavior from stock WebHDFS. At this point, it appears that the ADL protocol, while heavily inspired by the WebHDFS protocol, is not really a compatible match. It is its own protocol with its own unique requirements for clients to use it correctly and use it well. Accidentally connecting the ADL client to an HDFS cluster would be disastrous. The create/append/flush sequence would cause massive unsustainable load to the NameNode in terms of RPC calls and edit logging. Client write latency would be unacceptable. Likewise, accidentally connecting the stock WebHDFS client to ADL seems to yield unacceptable performance for ADL. It is these large deviations that lead me to conclude the best choice is a dedicated client distinct from the WebHDFS client code. Having full control of that client gives us the opportunity to provide the best possible user experience with ADL. As I've stated before though, I can accept a short-term plan of some code reuse with the WebHDFS client.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Apology Chris Nauroth, I was not aware of the protocol.

          Chris Nauroth, Chris Douglas, Aaron Fabbri, Tony Wu
          I am going to schedule a discussion with community on March 16, 2016 10:00 PM PST. I will share the details shortly on the channel so that everyone can join.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Apology Chris Nauroth , I was not aware of the protocol. Chris Nauroth , Chris Douglas , Aaron Fabbri , Tony Wu I am going to schedule a discussion with community on March 16, 2016 10:00 PM PST. I will share the details shortly on the channel so that everyone can join.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Incorporated review comment from the last discussion. Added synchronization logic for BufferManager as pointed out by Chris Douglas

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Incorporated review comment from the last discussion. Added synchronization logic for BufferManager as pointed out by Chris Douglas
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 8m 1s trunk passed
          +1 compile 7m 20s trunk passed with JDK v1.8.0_74
          +1 compile 7m 49s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 3m 18s trunk passed
          +1 mvneclipse 1m 7s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 2m 7s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 36s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 50s the patch passed
          +1 compile 6m 42s the patch passed with JDK v1.8.0_74
          +1 javac 6m 42s the patch passed
          +1 compile 7m 1s the patch passed with JDK v1.7.0_95
          +1 javac 7m 1s the patch passed
          +1 checkstyle 1m 8s the patch passed
          +1 mvnsite 3m 41s the patch passed
          +1 mvneclipse 1m 28s the patch passed
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 22s the patch passed
          +1 javadoc 2m 16s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 39s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74.
          +1 unit 7m 14s hadoop-common in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74.
          -1 unit 32m 39s hadoop-tools in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 11s hadoop-tools-dist in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.7.0_95.
          +1 unit 7m 20s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 unit 33m 32s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 13s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          149m 50s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.mapred.gridmix.TestRecordFactory



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793514/HADOOP-12666-009.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ccf59fdf5438 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / eba66a6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 8m 1s trunk passed +1 compile 7m 20s trunk passed with JDK v1.8.0_74 +1 compile 7m 49s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 8s trunk passed +1 mvnsite 3m 18s trunk passed +1 mvneclipse 1m 7s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 33s trunk passed +1 javadoc 2m 7s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 36s trunk passed with JDK v1.7.0_95 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 50s the patch passed +1 compile 6m 42s the patch passed with JDK v1.8.0_74 +1 javac 6m 42s the patch passed +1 compile 7m 1s the patch passed with JDK v1.7.0_95 +1 javac 7m 1s the patch passed +1 checkstyle 1m 8s the patch passed +1 mvnsite 3m 41s the patch passed +1 mvneclipse 1m 28s the patch passed -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 22s the patch passed +1 javadoc 2m 16s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 39s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_74. +1 unit 7m 14s hadoop-common in the patch passed with JDK v1.8.0_74. +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_74. -1 unit 32m 39s hadoop-tools in the patch failed with JDK v1.8.0_74. +1 unit 0m 11s hadoop-tools-dist in the patch passed with JDK v1.8.0_74. +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.7.0_95. +1 unit 7m 20s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 unit 33m 32s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 0m 13s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 149m 50s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.mapred.gridmix.TestRecordFactory Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793514/HADOOP-12666-009.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ccf59fdf5438 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / eba66a6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8849/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Rescheduling meeting to March 17, 2016 10:00 AM PST so that reviewer get enough time to review updated patch.

          Please join https://join.microsoft.com/meet/vdusane/SBLV5FRW

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Rescheduling meeting to March 17, 2016 10:00 AM PST so that reviewer get enough time to review updated patch. Please join https://join.microsoft.com/meet/vdusane/SBLV5FRW
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Participants: Chris Nauroth, Aaron Fabbri, Michael Han, Tony Wu, Chris Douglas, Vishwajeet Dusane, Shrikant Naidu.

          Meeting agenda

          • Discuss current comments and outstanding issues on HADOOP-12666
          • Approach of sub-classing webhdfs HDFS-9938

          Packaging:

          • Given the three options that were discussed in HDFS-9938, Chris N mentioned he was okay with the comments posted on the Jira. The option to maintain the current approach seems to be the best way forward in the short term. This is based on the understanding that the current approach is temporary and the client will evolve to be independent of WebHDFS. The Jira will be updated with this information.
          • The initial approach to use WebHDFS was a good starting point, but the reviewers feel that it is good to evolve the ADL client independent of webHDFS.
          • With the current approach changes in WebHDFS will impact the ADLS client.
          • The recommendation was to publish a long term plan of having a solution independent of WebHDFS and plan to target 2.9 for the separate ADL client (Long term plan)
          • Having such a plan would make the community more comfortable accepting the current solution.

          Create-Append Semantics:

          • Discussed overall create/append semantics, there was a concern raised that this does not ensure single writer semantics. .
          • Chris N did mention that this is a deviation from HDFS semantics of enforcing single writer semantics, and also stated that this is an approach taken by other cloud storage systems as well e.g WASB and s3.
          • There are some applications that do require this capability, typically these applications start writing to the same path, on recovery (e.g Hbase).
          • File Systems like WASB have made specific updates to address the needs of certain applications where handling multiple writers was an issue for e.g HBASE. WASB has implemented a specific lease implementation for HBASE
          • The ADL Client implementation also implements a similar lease semantic for Hbase and this is specifically done for createnonrecursive
            • It was clarified that the leaseid was generation by using a guid and there was an agreement on this approach
            • This information will be included in a separate document to be uploaded to the Jira (HADOOP-12666)
          • Chris N did mention that the general guideline for applications is to have each instance write data to its separate file and then commit by renaming it.
          • All accepted comments have been included in the latest patch
          • Buffer Manager Race condition - has been fixed in the latest patch.

          Contract test cases for HDFS do not implement ACL related test cases, since none of the file system extensions support them

            • Would need to create new contract tests for ACLs.
          • Overall across reviewers on the call there was no further objections to the core patches, reviewers plan to complete one more review of the updated patches.
          • HADOOP-12875 has been updated with a patch which includes an ability to run lives tests using contract tests, new test cases have been added:

          Followup Action items

          • Share/upload document that covers
            • information on read semantics, read ahead buffering, Create/Append semantics
            • Lease id generation to be included in the document
          • Share an overall plan on the roadmap for the ADL client - essentially what is the plan for removing the dependency on webhdfs (a "+1" on the Jira will be contingent on publishing this plan).Next step is for reviewers to complete the review of the new patch (Aaron to help with Cloudera reviewers)
          • Produce a page for alternative file systems
          • Attach Detailed documentation on file status cache (HADOOP-12876)
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Participants: Chris Nauroth , Aaron Fabbri , Michael Han , Tony Wu , Chris Douglas , Vishwajeet Dusane , Shrikant Naidu . Meeting agenda Discuss current comments and outstanding issues on HADOOP-12666 Approach of sub-classing webhdfs HDFS-9938 Packaging: Given the three options that were discussed in HDFS-9938 , Chris N mentioned he was okay with the comments posted on the Jira. The option to maintain the current approach seems to be the best way forward in the short term. This is based on the understanding that the current approach is temporary and the client will evolve to be independent of WebHDFS. The Jira will be updated with this information. The initial approach to use WebHDFS was a good starting point, but the reviewers feel that it is good to evolve the ADL client independent of webHDFS. With the current approach changes in WebHDFS will impact the ADLS client. The recommendation was to publish a long term plan of having a solution independent of WebHDFS and plan to target 2.9 for the separate ADL client (Long term plan) Having such a plan would make the community more comfortable accepting the current solution. Create-Append Semantics: Discussed overall create/append semantics, there was a concern raised that this does not ensure single writer semantics. . Chris N did mention that this is a deviation from HDFS semantics of enforcing single writer semantics, and also stated that this is an approach taken by other cloud storage systems as well e.g WASB and s3. There are some applications that do require this capability, typically these applications start writing to the same path, on recovery (e.g Hbase). File Systems like WASB have made specific updates to address the needs of certain applications where handling multiple writers was an issue for e.g HBASE. WASB has implemented a specific lease implementation for HBASE The ADL Client implementation also implements a similar lease semantic for Hbase and this is specifically done for createnonrecursive It was clarified that the leaseid was generation by using a guid and there was an agreement on this approach This information will be included in a separate document to be uploaded to the Jira ( HADOOP-12666 ) Chris N did mention that the general guideline for applications is to have each instance write data to its separate file and then commit by renaming it. All accepted comments have been included in the latest patch Buffer Manager Race condition - has been fixed in the latest patch. Contract test cases for HDFS do not implement ACL related test cases, since none of the file system extensions support them Would need to create new contract tests for ACLs. Overall across reviewers on the call there was no further objections to the core patches, reviewers plan to complete one more review of the updated patches. HADOOP-12875 has been updated with a patch which includes an ability to run lives tests using contract tests, new test cases have been added: Followup Action items Share/upload document that covers information on read semantics, read ahead buffering, Create/Append semantics Lease id generation to be included in the document Share an overall plan on the roadmap for the ADL client - essentially what is the plan for removing the dependency on webhdfs (a "+1" on the Jira will be contingent on publishing this plan).Next step is for reviewers to complete the review of the new patch (Aaron to help with Cloudera reviewers) Produce a page for alternative file systems Documents the differences to HDFS ; Example: HADOOP-12635 Attach Detailed documentation on file status cache ( HADOOP-12876 )
          Hide
          twu Tony Wu added a comment -

          Hi Vishwajeet Dusane,

          Thank you for posting the semantics document, meeting minutes and a new patch. It was really helpful.

          General comments:

          1. Regarding the semantics document. It will be great if you can also include more information on how ADL backend can "lock" a file for write. From the doc and our meeting discussion, there seems to be 2 ways (please confirm):
            1. File lease (which you included in the doc). Used by createNonRecursive()
            2. Maintain connection to the backend (you mentioned during meeting). Used by append()
              • In this case do we assume ADL backend tracks which file is opened for write by keeping track of HTTP connections to the file?
          2. In hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md. You mentioned:

            User and group information returned as ListStatus and GetFileStatus is in form of GUID associated in Azure Active Directory.

            There are applications which verifies the file ownership and it will fail because of this. Also I believe this means commands like hdfs dfs -ls <some path> will return GUID as user & group. It may not be that readable. Can you comment on how users should handle this?

            • I see there is a adl.debug.override.localuserasfileowner config which will override the user with local client user and group with "hdfs". But this workaround is probably not for actual usage.

          Specific comments:

          1. Can you explain why flushAsync() is used in BatchAppendOutputStream? It seems like flushAsync() is only used in a particular case where there is some data left in the buffer from previous writes and that combined with the current write will cross the buffer boundary. I'm not sure why this particular flush has to be async.

          Consider the following case:

          1. BatchAppendOutputStream#flushAsync() returns. flushAsync() sends the sync job to thread and sets offset to 0:
                private void flushAsync() throws IOException {
                  if (offset > 0) {
                    waitForOutstandingFlush();
            
                    // Submit the new flush task to the executor
                    flushTask = EXECUTOR.submit(new CommitTask(data, offset, eof));
            
                    // Get a new internal buffer for the user to write
                    data = getBuffer();
                    offset = 0;
                  }
                }
            
          2. BatchAppendOutputStream#write() returns.
          3. Client closes outputStream.
          4. BatchAppendOutputStream#close() checks to see if there's anything to flush by checking offset. And in this case there is nothing to flush because offset is set to 0 earlier.
                  boolean flushedSomething = false;
                  if (hadError) {
                    // No point proceeding further since the error has occurred and
                    // stream would be required to upload again.
                    return;
                  } else {
                    flushedSomething = offset > 0;
                    flush();
                  }
            
          5. BatchAppendOutputStream#close() does not wait for the async flush job to complete. After this point if flushAsync() hit any error, this error will be lost and client will not be aware of it.
          6. If client then starts to write (append) to the same file with a new stream, this new write also will not wait for the previous async job to complete because flushTask is internal to BatchAppendOutputStream. It might be possible for the 2 writes to reach the backend in reverse order.

          IMHO if this async flush is necessary then EXECUTOR should be created inside BatchAppendOutputStream and shutdown when the stream is closed. Currently EXECUTOR lives in PrivateAzureDataLakeFileSystem.

          2. EXECUTOR in PrivateAzureDataLakeFileSystem is not shutdown properly.

          3. Stream is closed check is missing in BatchAppendOutputStream. This check is present for BatchByteArrayInputStream.

          Show
          twu Tony Wu added a comment - Hi Vishwajeet Dusane , Thank you for posting the semantics document, meeting minutes and a new patch. It was really helpful. General comments: Regarding the semantics document. It will be great if you can also include more information on how ADL backend can "lock" a file for write. From the doc and our meeting discussion, there seems to be 2 ways (please confirm): File lease (which you included in the doc). Used by createNonRecursive() Maintain connection to the backend (you mentioned during meeting). Used by append() In this case do we assume ADL backend tracks which file is opened for write by keeping track of HTTP connections to the file? In hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md . You mentioned: User and group information returned as ListStatus and GetFileStatus is in form of GUID associated in Azure Active Directory. There are applications which verifies the file ownership and it will fail because of this. Also I believe this means commands like hdfs dfs -ls <some path> will return GUID as user & group. It may not be that readable. Can you comment on how users should handle this? I see there is a adl.debug.override.localuserasfileowner config which will override the user with local client user and group with "hdfs". But this workaround is probably not for actual usage. Specific comments: 1. Can you explain why flushAsync() is used in BatchAppendOutputStream ? It seems like flushAsync() is only used in a particular case where there is some data left in the buffer from previous writes and that combined with the current write will cross the buffer boundary. I'm not sure why this particular flush has to be async. Consider the following case: BatchAppendOutputStream#flushAsync() returns. flushAsync() sends the sync job to thread and sets offset to 0: private void flushAsync() throws IOException { if (offset > 0) { waitForOutstandingFlush(); // Submit the new flush task to the executor flushTask = EXECUTOR.submit( new CommitTask(data, offset, eof)); // Get a new internal buffer for the user to write data = getBuffer(); offset = 0; } } BatchAppendOutputStream#write() returns. Client closes outputStream. BatchAppendOutputStream#close() checks to see if there's anything to flush by checking offset. And in this case there is nothing to flush because offset is set to 0 earlier. boolean flushedSomething = false ; if (hadError) { // No point proceeding further since the error has occurred and // stream would be required to upload again. return ; } else { flushedSomething = offset > 0; flush(); } BatchAppendOutputStream#close() does not wait for the async flush job to complete. After this point if flushAsync() hit any error, this error will be lost and client will not be aware of it. If client then starts to write (append) to the same file with a new stream, this new write also will not wait for the previous async job to complete because flushTask is internal to BatchAppendOutputStream . It might be possible for the 2 writes to reach the backend in reverse order. IMHO if this async flush is necessary then EXECUTOR should be created inside BatchAppendOutputStream and shutdown when the stream is closed. Currently EXECUTOR lives in PrivateAzureDataLakeFileSystem . 2. EXECUTOR in PrivateAzureDataLakeFileSystem is not shutdown properly. 3. Stream is closed check is missing in BatchAppendOutputStream . This check is present for BatchByteArrayInputStream .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Tony Wu Thank you for the comments.

          Regarding the semantics document. It will be great if you can also include more information on how ADL backend can "lock" a file for write....

          Like we discussed and covered in MOM shared on the JIRA. Similar to WASB, Adl client also implements "lock" for write during createnonrecursive. FileSystem append and create calls does not lock a file. Lease id mechanism ensures "lock" over a file. Similar approach is taken by WASB and S3. Same is covered in the semantics document as well. Please let me know if i misread your question.

          User and group information returned as ListStatus and GetFileStatus is in form of GUID associated in Azure Active Directory.

          Ongoing effort at the moment to support UPN instead of GUID.

          Skipping all the questions/comments on flushAsync, since i am already working on better append algorithm. Best would be, i will nuke the flushAsync implementation from this review to avoid any confusion. Would raise a separate JIRA when fast append feature is ready.

          Stream is closed check is missing in BatchAppendOutputStream. This check is present for BatchByteArrayInputStream.

          Corner case however i will update to have close check BatchAppendOutputStream as well.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Tony Wu Thank you for the comments. Regarding the semantics document. It will be great if you can also include more information on how ADL backend can "lock" a file for write.... Like we discussed and covered in MOM shared on the JIRA. Similar to WASB, Adl client also implements "lock" for write during createnonrecursive. FileSystem append and create calls does not lock a file. Lease id mechanism ensures "lock" over a file. Similar approach is taken by WASB and S3. Same is covered in the semantics document as well. Please let me know if i misread your question. User and group information returned as ListStatus and GetFileStatus is in form of GUID associated in Azure Active Directory. Ongoing effort at the moment to support UPN instead of GUID. Skipping all the questions/comments on flushAsync, since i am already working on better append algorithm. Best would be, i will nuke the flushAsync implementation from this review to avoid any confusion. Would raise a separate JIRA when fast append feature is ready. Stream is closed check is missing in BatchAppendOutputStream. This check is present for BatchByteArrayInputStream. Corner case however i will update to have close check BatchAppendOutputStream as well.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Nauroth, Chris Douglas, Aaron Fabbri, Sean Mackrory and Tony Wu - Thanks a lot guys for being active on the review so far. To move this review forward, Could you please share if you have any more review comments and if not, could you please at least your status on the review? This would help us to estimate the effort required.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Nauroth , Chris Douglas , Aaron Fabbri , Sean Mackrory and Tony Wu - Thanks a lot guys for being active on the review so far. To move this review forward, Could you please share if you have any more review comments and if not, could you please at least your status on the review? This would help us to estimate the effort required.
          Hide
          cnauroth Chris Nauroth added a comment -

          Vishwajeet Dusane, thank you for patch v009. It looks like this has mostly addressed my prior feedback.

          I see one more mishandled InterruptedException inside BatchByteArrayInputStream. That would be good to clean up.

          We still have an open question about LOG_VERSION. Here is an earlier comment from me:

          Can you please elaborate on what LOG_VERSION is supposed to mean? Does "code instrumentation version" mean that this is trying to capture the version of the Hadoop software that is running, or is there some special significance to the hard-coded "1.2.1" value? If it's meant to indicate the Hadoop software version, then the Hadoop VersionInfo class I mentioned would be a good fit.

          This was your response:

          LOG_VERSION is telemetry information used in Adl back-end only.

          This didn't entirely address the question for me. Does this mean that the LOG_VERSION is an opaque value used by telemetry only for tracking the "client version" that sent the request? If so, then my suggestion to use the Hadoop VersionInfo class would be more accurate. If not, then can you help me better understand the significance of the current hard-coded 1.2.1 value? Does the service telemetry have some kind of dependency on that specific value?

          One thing I would want to avoid is a situation where this value needs to be maintained in lock step with something in the back-end implementation.

          Show
          cnauroth Chris Nauroth added a comment - Vishwajeet Dusane , thank you for patch v009. It looks like this has mostly addressed my prior feedback. I see one more mishandled InterruptedException inside BatchByteArrayInputStream . That would be good to clean up. We still have an open question about LOG_VERSION . Here is an earlier comment from me: Can you please elaborate on what LOG_VERSION is supposed to mean? Does "code instrumentation version" mean that this is trying to capture the version of the Hadoop software that is running, or is there some special significance to the hard-coded "1.2.1" value? If it's meant to indicate the Hadoop software version, then the Hadoop VersionInfo class I mentioned would be a good fit. This was your response: LOG_VERSION is telemetry information used in Adl back-end only. This didn't entirely address the question for me. Does this mean that the LOG_VERSION is an opaque value used by telemetry only for tracking the "client version" that sent the request? If so, then my suggestion to use the Hadoop VersionInfo class would be more accurate. If not, then can you help me better understand the significance of the current hard-coded 1.2.1 value? Does the service telemetry have some kind of dependency on that specific value? One thing I would want to avoid is a situation where this value needs to be maintained in lock step with something in the back-end implementation.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -
          • Removed Asynchronous append operation.
          • For version tracking, added usage of Hadoop VersionInfo.
          • Reduced buffer caching size from 16MB to 8 MB. Observed during few HBase runs like snapshot creation. Aggressive buffering does not help due to random positional reads in the file.
          • Removed LOG_VERSION usage since telemetry is not part of this patch set anyway,
          • Added a check for stream closed status for BatchAppendOutputStream.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Removed Asynchronous append operation. For version tracking, added usage of Hadoop VersionInfo . Reduced buffer caching size from 16MB to 8 MB. Observed during few HBase runs like snapshot creation. Aggressive buffering does not help due to random positional reads in the file. Removed LOG_VERSION usage since telemetry is not part of this patch set anyway, Added a check for stream closed status for BatchAppendOutputStream .
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks Chris Nauroth

          • Incorporated your comment for InterruptedException ...
          • For LOG_VERSION, was mainly used during logging was enabled to differentiate on the cluster if the adl driver is running with which change set. I have removed usage of LOG_VERSION and incorporated VersionInfo since it helps to differentiate the same. Version information does not have any dependency on the service telemetry.
          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks Chris Nauroth Incorporated your comment for InterruptedException ... For LOG_VERSION , was mainly used during logging was enabled to differentiate on the cluster if the adl driver is running with which change set. I have removed usage of LOG_VERSION and incorporated VersionInfo since it helps to differentiate the same. Version information does not have any dependency on the service telemetry.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 6m 52s trunk passed with JDK v1.8.0_77
          +1 compile 7m 12s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 4m 22s trunk passed
          +1 mvneclipse 1m 7s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 3m 38s trunk passed
          +1 javadoc 3m 20s trunk passed with JDK v1.8.0_77
          +1 javadoc 4m 23s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 3m 22s the patch passed
          +1 compile 7m 39s the patch passed with JDK v1.8.0_77
          +1 javac 7m 39s the patch passed
          +1 compile 7m 26s the patch passed with JDK v1.7.0_95
          +1 javac 7m 26s the patch passed
          +1 checkstyle 1m 9s the patch passed
          +1 mvnsite 4m 36s the patch passed
          +1 mvneclipse 1m 27s the patch passed
          +1 shellcheck 0m 10s The applied patch generated 0 new + 92 unchanged - 1 fixed = 92 total (was 93)
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 5m 0s the patch passed
          +1 javadoc 3m 47s the patch passed with JDK v1.8.0_77
          +1 javadoc 4m 55s the patch passed with JDK v1.7.0_95
          +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_77.
          +1 unit 9m 54s hadoop-common in the patch passed with JDK v1.8.0_77.
          -1 unit 68m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          +1 unit 0m 15s hadoop-azure-datalake in the patch passed with JDK v1.8.0_77.
          +1 unit 36m 14s hadoop-tools in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 14s hadoop-project in the patch passed with JDK v1.7.0_95.
          -1 unit 9m 51s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 64m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 22s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 unit 34m 11s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 35s Patch does not generate ASF License warnings.
          305m 17s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestDecommission
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestDataTransferProtocol
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.fs.contract.hdfs.TestHDFSContractConcat
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.TestLargeBlock
            hadoop.hdfs.TestDFSUpgradeFromImage
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798254/HADOOP-12666-010.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 2d3861ba14d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 209303b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 6m 52s trunk passed with JDK v1.8.0_77 +1 compile 7m 12s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 4m 22s trunk passed +1 mvneclipse 1m 7s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 3m 38s trunk passed +1 javadoc 3m 20s trunk passed with JDK v1.8.0_77 +1 javadoc 4m 23s trunk passed with JDK v1.7.0_95 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 3m 22s the patch passed +1 compile 7m 39s the patch passed with JDK v1.8.0_77 +1 javac 7m 39s the patch passed +1 compile 7m 26s the patch passed with JDK v1.7.0_95 +1 javac 7m 26s the patch passed +1 checkstyle 1m 9s the patch passed +1 mvnsite 4m 36s the patch passed +1 mvneclipse 1m 27s the patch passed +1 shellcheck 0m 10s The applied patch generated 0 new + 92 unchanged - 1 fixed = 92 total (was 93) -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 5m 0s the patch passed +1 javadoc 3m 47s the patch passed with JDK v1.8.0_77 +1 javadoc 4m 55s the patch passed with JDK v1.7.0_95 +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_77. +1 unit 9m 54s hadoop-common in the patch passed with JDK v1.8.0_77. -1 unit 68m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_77. +1 unit 0m 15s hadoop-azure-datalake in the patch passed with JDK v1.8.0_77. +1 unit 36m 14s hadoop-tools in the patch passed with JDK v1.8.0_77. +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.8.0_77. +1 unit 0m 14s hadoop-project in the patch passed with JDK v1.7.0_95. -1 unit 9m 51s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 64m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 unit 0m 22s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 unit 34m 11s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 305m 17s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestDecommission   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.TestReadStripedFileWithMissingBlocks JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestDecommission   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.TestDFSUpgradeFromImage JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798254/HADOOP-12666-010.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 2d3861ba14d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 209303b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9063/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          -1 for whitespace is not caused due to this patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - -1 for whitespace is not caused due to this patch.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Nauroth, Chris Douglas, Tony Wu, Aaron Fabbri and Sean Mackrory - Could you please let me know if you have any more comments that is not addressed so far ? Thanks a lot again for the review.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Nauroth , Chris Douglas , Tony Wu , Aaron Fabbri and Sean Mackrory - Could you please let me know if you have any more comments that is not addressed so far ? Thanks a lot again for the review.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          1. Removed unchanged file hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          2. Marked configuration function as VisibleForTesting as they are opened for testing purpose only.
          3. Removed overridden API which is just invoking super method.
          4. Reduced to OkHttp dependency from OkHttp3 to match Hadoop build expectation.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - 1. Removed unchanged file hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs 2. Marked configuration function as VisibleForTesting as they are opened for testing purpose only. 3. Removed overridden API which is just invoking super method. 4. Reduced to OkHttp dependency from OkHttp3 to match Hadoop build expectation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 22s Maven dependency ordering for branch
          +1 mvninstall 6m 52s trunk passed
          +1 compile 6m 13s trunk passed with JDK v1.8.0_91
          +1 compile 6m 48s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 25s trunk passed
          +1 mvnsite 3m 10s trunk passed
          +1 mvneclipse 1m 10s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 2m 1s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 23s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 30s the patch passed
          +1 compile 5m 58s the patch passed with JDK v1.8.0_91
          +1 javac 5m 58s the patch passed
          +1 compile 6m 44s the patch passed with JDK v1.7.0_95
          +1 javac 6m 44s the patch passed
          +1 checkstyle 1m 24s the patch passed
          +1 mvnsite 3m 30s the patch passed
          +1 mvneclipse 1m 16s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 27s the patch passed
          +1 javadoc 2m 12s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 42s the patch passed with JDK v1.7.0_95
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_91.
          -1 unit 7m 47s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91.
          +1 unit 32m 29s hadoop-tools in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-tools-dist in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.7.0_95.
          -1 unit 7m 52s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 16s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 unit 33m 31s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          146m 57s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.net.TestDNS
            hadoop.conf.TestCommonConfigurationFields
          JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803228/HADOOP-12666-011.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c20da6ef3c94 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 87f5e35
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 22s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 6m 13s trunk passed with JDK v1.8.0_91 +1 compile 6m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 25s trunk passed +1 mvnsite 3m 10s trunk passed +1 mvneclipse 1m 10s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 31s trunk passed +1 javadoc 2m 1s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 23s trunk passed with JDK v1.7.0_95 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 30s the patch passed +1 compile 5m 58s the patch passed with JDK v1.8.0_91 +1 javac 5m 58s the patch passed +1 compile 6m 44s the patch passed with JDK v1.7.0_95 +1 javac 6m 44s the patch passed +1 checkstyle 1m 24s the patch passed +1 mvnsite 3m 30s the patch passed +1 mvneclipse 1m 16s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 27s the patch passed +1 javadoc 2m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 42s the patch passed with JDK v1.7.0_95 +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_91. -1 unit 7m 47s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 11s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91. +1 unit 32m 29s hadoop-tools in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-tools-dist in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.7.0_95. -1 unit 7m 52s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 16s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 unit 33m 31s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 0m 15s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 146m 57s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.net.TestDNS   hadoop.conf.TestCommonConfigurationFields JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803228/HADOOP-12666-011.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c20da6ef3c94 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 87f5e35 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9352/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          CachedRefreshTokenBasedAccessTokenProvider

          • Since the AccessTokenProvider is only created by reflection, the Timer cstr is for testing and does not require an override in this subclass
          • The static instance should be final and created during class initialization, but...
          • ConfRefreshTokenBasedAccessTokenProvider is not threadsafe. setConf will update the static instance without synchronization, which is shared by every instance of CachedRTBATP. This could cause undefined behavior. The intent is to be to pool clients with the same parameters? Would it make sense to add a small cache (v12)?

          PrivateCachedRefreshTokenBasedAccessTokenProvider

          • The override doesn't seem to serve a purpose. Since it's a workaround, adding audience/visibility annotations (HADOOP-5073) would emphasize that this is temporary.

          PrivateAzureDataLakeFileSystem

          • catching ArrayIndexOutOfBoundsException instead of performing proper bounds checking in BufferManager::get is not efficient:
            PrivateAzureDataLakeFileSystem.java
            synchronized (BufferManager.getLock()) {
              if (bm.hasData(fsPath.toString(), fileOffset, len)) {
                try {
                  bm.get(data, fileOffset);
                  validDataHoldingSize = data.length;
                  currentFileOffset = fileOffset;
                } catch (ArrayIndexOutOfBoundsException e) {
                  fetchDataOverNetwork = true;
                }
              } else {
                fetchDataOverNetwork = true;
              }
            }
            
            BufferManager.java
            void get(byte[] data, long offset) {
              System.arraycopy(buffer.data, (int) (offset - buffer.offset), data, 0,
                  data.length);
            }
            

          The BufferManager/PrivateAzureDataLakeFileSystem synchronization is unorthodox, and verifying its correctness is not straightforward. Layering that complexity on top of the readahead logic without simplifying abstractions makes it very difficult to review. I hope subsequent revisions will replace this code with a clearer model, because the current code will be very difficult to maintain.

          Show
          chris.douglas Chris Douglas added a comment - CachedRefreshTokenBasedAccessTokenProvider Since the AccessTokenProvider is only created by reflection, the Timer cstr is for testing and does not require an override in this subclass The static instance should be final and created during class initialization, but... ConfRefreshTokenBasedAccessTokenProvider is not threadsafe. setConf will update the static instance without synchronization, which is shared by every instance of CachedRTBATP . This could cause undefined behavior. The intent is to be to pool clients with the same parameters? Would it make sense to add a small cache (v12)? PrivateCachedRefreshTokenBasedAccessTokenProvider The override doesn't seem to serve a purpose. Since it's a workaround, adding audience/visibility annotations ( HADOOP-5073 ) would emphasize that this is temporary. PrivateAzureDataLakeFileSystem catching ArrayIndexOutOfBoundsException instead of performing proper bounds checking in BufferManager::get is not efficient: PrivateAzureDataLakeFileSystem.java synchronized (BufferManager.getLock()) { if (bm.hasData(fsPath.toString(), fileOffset, len)) { try { bm.get(data, fileOffset); validDataHoldingSize = data.length; currentFileOffset = fileOffset; } catch (ArrayIndexOutOfBoundsException e) { fetchDataOverNetwork = true ; } } else { fetchDataOverNetwork = true ; } } BufferManager.java void get( byte [] data, long offset) { System .arraycopy(buffer.data, ( int ) (offset - buffer.offset), data, 0, data.length); } The BufferManager/PrivateAzureDataLakeFileSystem synchronization is unorthodox, and verifying its correctness is not straightforward. Layering that complexity on top of the readahead logic without simplifying abstractions makes it very difficult to review. I hope subsequent revisions will replace this code with a clearer model, because the current code will be very difficult to maintain.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 49s Maven dependency ordering for branch
          +1 mvninstall 7m 23s trunk passed
          +1 compile 5m 46s trunk passed with JDK v1.8.0_91
          +1 compile 6m 37s trunk passed with JDK v1.7.0_101
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 3m 14s trunk passed
          +1 mvneclipse 1m 54s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 1m 57s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 21s trunk passed with JDK v1.7.0_101
          0 mvndep 0m 7s Maven dependency ordering for patch
          -1 mvninstall 0m 7s hadoop-tools in the patch failed.
          -1 mvninstall 0m 5s hadoop-azure-datalake in the patch failed.
          -1 mvninstall 0m 8s hadoop-tools-dist in the patch failed.
          -1 compile 0m 6s root in the patch failed with JDK v1.8.0_91.
          -1 javac 0m 6s root in the patch failed with JDK v1.8.0_91.
          -1 compile 0m 7s root in the patch failed with JDK v1.7.0_101.
          -1 javac 0m 7s root in the patch failed with JDK v1.7.0_101.
          +1 checkstyle 0m 7s the patch passed
          -1 mvnsite 0m 6s hadoop-tools in the patch failed.
          -1 mvnsite 0m 6s hadoop-azure-datalake in the patch failed.
          -1 mvnsite 0m 7s hadoop-tools-dist in the patch failed.
          -1 mvneclipse 0m 6s hadoop-tools in the patch failed.
          -1 mvneclipse 0m 6s hadoop-azure-datalake in the patch failed.
          -1 mvneclipse 0m 8s hadoop-tools-dist in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          -1 findbugs 0m 5s hadoop-azure-datalake in the patch failed.
          -1 javadoc 0m 5s hadoop-tools in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 6s hadoop-tools-dist in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 7s hadoop-tools in the patch failed with JDK v1.7.0_101.
          -1 javadoc 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101.
          -1 javadoc 0m 7s hadoop-tools-dist in the patch failed with JDK v1.7.0_101.
          -1 unit 6m 23s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.8.0_91.
          -1 unit 0m 6s hadoop-tools in the patch failed with JDK v1.8.0_91.
          -1 unit 0m 4s hadoop-azure-datalake in the patch failed with JDK v1.8.0_91.
          -1 unit 0m 7s hadoop-tools-dist in the patch failed with JDK v1.8.0_91.
          -1 unit 6m 42s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.7.0_101.
          -1 unit 0m 7s hadoop-tools in the patch failed with JDK v1.7.0_101.
          -1 unit 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101.
          -1 unit 0m 8s hadoop-tools-dist in the patch failed with JDK v1.7.0_101.
          0 asflicense 0m 8s ASF License check generated no output?
          58m 37s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.conf.TestCommonConfigurationFields
          JDK v1.7.0_101 Failed junit tests hadoop.conf.TestCommonConfigurationFields
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803990/HADOOP-12666-012.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 6189a31c8942 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fa1380
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-tools-dist.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.8.0_91.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.8.0_91.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-tools-dist.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-azure-datalake.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-tools-dist.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.7.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 49s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 5m 46s trunk passed with JDK v1.8.0_91 +1 compile 6m 37s trunk passed with JDK v1.7.0_101 +1 checkstyle 1m 27s trunk passed +1 mvnsite 3m 14s trunk passed +1 mvneclipse 1m 54s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 39s trunk passed +1 javadoc 1m 57s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 21s trunk passed with JDK v1.7.0_101 0 mvndep 0m 7s Maven dependency ordering for patch -1 mvninstall 0m 7s hadoop-tools in the patch failed. -1 mvninstall 0m 5s hadoop-azure-datalake in the patch failed. -1 mvninstall 0m 8s hadoop-tools-dist in the patch failed. -1 compile 0m 6s root in the patch failed with JDK v1.8.0_91. -1 javac 0m 6s root in the patch failed with JDK v1.8.0_91. -1 compile 0m 7s root in the patch failed with JDK v1.7.0_101. -1 javac 0m 7s root in the patch failed with JDK v1.7.0_101. +1 checkstyle 0m 7s the patch passed -1 mvnsite 0m 6s hadoop-tools in the patch failed. -1 mvnsite 0m 6s hadoop-azure-datalake in the patch failed. -1 mvnsite 0m 7s hadoop-tools-dist in the patch failed. -1 mvneclipse 0m 6s hadoop-tools in the patch failed. -1 mvneclipse 0m 6s hadoop-azure-datalake in the patch failed. -1 mvneclipse 0m 8s hadoop-tools-dist in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist -1 findbugs 0m 5s hadoop-azure-datalake in the patch failed. -1 javadoc 0m 5s hadoop-tools in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 6s hadoop-tools-dist in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 7s hadoop-tools in the patch failed with JDK v1.7.0_101. -1 javadoc 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101. -1 javadoc 0m 7s hadoop-tools-dist in the patch failed with JDK v1.7.0_101. -1 unit 6m 23s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.8.0_91. -1 unit 0m 6s hadoop-tools in the patch failed with JDK v1.8.0_91. -1 unit 0m 4s hadoop-azure-datalake in the patch failed with JDK v1.8.0_91. -1 unit 0m 7s hadoop-tools-dist in the patch failed with JDK v1.8.0_91. -1 unit 6m 42s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.7.0_101. -1 unit 0m 7s hadoop-tools in the patch failed with JDK v1.7.0_101. -1 unit 0m 5s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101. -1 unit 0m 8s hadoop-tools-dist in the patch failed with JDK v1.7.0_101. 0 asflicense 0m 8s ASF License check generated no output? 58m 37s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.conf.TestCommonConfigurationFields JDK v1.7.0_101 Failed junit tests hadoop.conf.TestCommonConfigurationFields   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803990/HADOOP-12666-012.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 6189a31c8942 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa1380 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-tools-dist.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.8.0_91.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.8.0_91.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-tools-dist.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-azure-datalake.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-tools-dist.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.7.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9421/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -
          [ERROR] The build could not read 1 project -> [Help 1]
          [ERROR]   
          [ERROR]   The project org.apache.hadoop:hadoop-azure-datalake:3.0.0-SNAPSHOT (/testptch/hadoop/hadoop-tools/hadoop-azure-datalake/pom.xml) has 1 error
          [ERROR]     Non-resolvable parent POM: Could not find artifact org.apache.hadoop:hadoop-project:pom:3.0.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 19, column 11 -> [Help 2]
          

          This didn't change from v11. Will resubmit.

          Show
          chris.douglas Chris Douglas added a comment - [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.hadoop:hadoop-azure-datalake:3.0.0-SNAPSHOT (/testptch/hadoop/hadoop-tools/hadoop-azure-datalake/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.apache.hadoop:hadoop-project:pom:3.0.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 19, column 11 -> [Help 2] This didn't change from v11. Will resubmit.
          Hide
          cnauroth Chris Nauroth added a comment -

          This didn't change from v11. Will resubmit.

          Trunk recently changed the version number from 3.0.0-SNAPSHOT to 3.0.0-alpha1-SNAPSHOT in preparation for building 3.x release candidates. We'll need to update the version number in this patch's pom.xml to match. I'm uploading patch v013 with just that one-line change.

          Show
          cnauroth Chris Nauroth added a comment - This didn't change from v11. Will resubmit. Trunk recently changed the version number from 3.0.0-SNAPSHOT to 3.0.0-alpha1-SNAPSHOT in preparation for building 3.x release candidates. We'll need to update the version number in this patch's pom.xml to match. I'm uploading patch v013 with just that one-line change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 45s trunk passed
          +1 compile 7m 49s trunk passed with JDK v1.8.0_91
          +1 compile 7m 26s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 3m 21s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
          +1 findbugs 1m 36s trunk passed
          +1 javadoc 2m 12s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 39s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 2m 47s the patch passed
          +1 compile 7m 8s the patch passed with JDK v1.8.0_91
          +1 javac 7m 8s the patch passed
          +1 compile 7m 25s the patch passed with JDK v1.7.0_95
          +1 javac 7m 25s the patch passed
          -1 checkstyle 1m 23s root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 3m 31s the patch passed
          +1 mvneclipse 1m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 34s the patch passed
          +1 javadoc 2m 26s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91.
          -1 unit 9m 6s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 14s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91.
          +1 unit 34m 35s hadoop-tools in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 10s hadoop-tools-dist in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_95.
          -1 unit 8m 43s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 17s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95.
          +1 unit 33m 44s hadoop-tools in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 13s hadoop-tools-dist in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          156m 34s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.conf.TestCommonConfigurationFields
          JDK v1.7.0_95 Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804027/HADOOP-12666-013.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 0c2095c12f87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fa1380
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 45s trunk passed +1 compile 7m 49s trunk passed with JDK v1.8.0_91 +1 compile 7m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 26s trunk passed +1 mvnsite 3m 21s trunk passed +1 mvneclipse 0m 59s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools +1 findbugs 1m 36s trunk passed +1 javadoc 2m 12s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 39s trunk passed with JDK v1.7.0_95 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 2m 47s the patch passed +1 compile 7m 8s the patch passed with JDK v1.8.0_91 +1 javac 7m 8s the patch passed +1 compile 7m 25s the patch passed with JDK v1.7.0_95 +1 javac 7m 25s the patch passed -1 checkstyle 1m 23s root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 3m 31s the patch passed +1 mvneclipse 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 34s the patch passed +1 javadoc 2m 26s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91. -1 unit 9m 6s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 14s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91. +1 unit 34m 35s hadoop-tools in the patch passed with JDK v1.8.0_91. +1 unit 0m 10s hadoop-tools-dist in the patch passed with JDK v1.8.0_91. +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_95. -1 unit 8m 43s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 17s hadoop-azure-datalake in the patch passed with JDK v1.7.0_95. +1 unit 33m 44s hadoop-tools in the patch passed with JDK v1.7.0_95. +1 unit 0m 13s hadoop-tools-dist in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 156m 34s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.conf.TestCommonConfigurationFields JDK v1.7.0_95 Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804027/HADOOP-12666-013.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 0c2095c12f87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa1380 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9426/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Thanks, Chris Nauroth.

          The checkstyle warning is being addressed in HADOOP-13160. Updated the patch to fix the test for HADOOP-12738.

          Show
          chris.douglas Chris Douglas added a comment - Thanks, Chris Nauroth . The checkstyle warning is being addressed in HADOOP-13160 . Updated the patch to fix the test for HADOOP-12738 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          -1 mvninstall 0m 7s root in trunk failed.
          +1 compile 6m 30s trunk passed with JDK v1.8.0_91
          -1 compile 0m 10s root in trunk failed with JDK v1.7.0_101.
          +1 checkstyle 1m 50s trunk passed
          -1 mvnsite 0m 18s hadoop-common in trunk failed.
          -1 mvnsite 0m 15s hadoop-tools in trunk failed.
          -1 mvneclipse 0m 9s hadoop-common in trunk failed.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          -1 findbugs 0m 8s hadoop-common in trunk failed.
          -1 javadoc 0m 8s hadoop-common in trunk failed with JDK v1.8.0_91.
          -1 javadoc 0m 9s hadoop-common in trunk failed with JDK v1.7.0_101.
          0 mvndep 0m 10s Maven dependency ordering for patch
          -1 mvninstall 0m 8s hadoop-common in the patch failed.
          -1 mvninstall 0m 10s hadoop-project in the patch failed.
          -1 mvninstall 0m 19s hadoop-tools in the patch failed.
          -1 mvninstall 0m 11s hadoop-azure-datalake in the patch failed.
          -1 mvninstall 0m 10s hadoop-tools-dist in the patch failed.
          +1 compile 6m 10s the patch passed with JDK v1.8.0_91
          +1 javac 6m 10s the patch passed
          -1 compile 0m 10s root in the patch failed with JDK v1.7.0_101.
          -1 javac 0m 10s root in the patch failed with JDK v1.7.0_101.
          -1 checkstyle 1m 25s root: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
          -1 mvnsite 0m 12s hadoop-common in the patch failed.
          -1 mvnsite 0m 12s hadoop-tools in the patch failed.
          -1 mvnsite 0m 12s hadoop-azure-datalake in the patch failed.
          -1 mvnsite 0m 10s hadoop-tools-dist in the patch failed.
          -1 mvneclipse 0m 9s hadoop-common in the patch failed.
          -1 mvneclipse 0m 24s hadoop-tools in the patch failed.
          -1 mvneclipse 0m 10s hadoop-tools-dist in the patch failed.
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          -1 findbugs 0m 8s hadoop-common in the patch failed.
          -1 findbugs 0m 11s hadoop-azure-datalake in the patch failed.
          -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 45s hadoop-tools in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 7s hadoop-tools-dist in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.7.0_101.
          -1 javadoc 0m 58s hadoop-tools in the patch failed with JDK v1.7.0_101.
          -1 javadoc 0m 9s hadoop-tools-dist in the patch failed with JDK v1.7.0_101.
          -1 unit 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_91.
          -1 unit 32m 26s hadoop-tools in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 16s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91.
          -1 unit 0m 10s hadoop-tools-dist in the patch failed with JDK v1.8.0_91.
          -1 unit 0m 10s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_101.
          -1 unit 0m 18s hadoop-tools in the patch failed with JDK v1.7.0_101.
          -1 unit 0m 11s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101.
          -1 unit 0m 11s hadoop-tools-dist in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          63m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804300/HADOOP-12666-014.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 216c5763accf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6a6e74a
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-compile-root-jdk1.7.0_101.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvnsite-hadoop-tools.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvneclipse-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-tools-dist.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-tools-dist.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-common-project_hadoop-common.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-tools.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-tools-dist.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.7.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch -1 mvninstall 0m 7s root in trunk failed. +1 compile 6m 30s trunk passed with JDK v1.8.0_91 -1 compile 0m 10s root in trunk failed with JDK v1.7.0_101. +1 checkstyle 1m 50s trunk passed -1 mvnsite 0m 18s hadoop-common in trunk failed. -1 mvnsite 0m 15s hadoop-tools in trunk failed. -1 mvneclipse 0m 9s hadoop-common in trunk failed. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist -1 findbugs 0m 8s hadoop-common in trunk failed. -1 javadoc 0m 8s hadoop-common in trunk failed with JDK v1.8.0_91. -1 javadoc 0m 9s hadoop-common in trunk failed with JDK v1.7.0_101. 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 0m 8s hadoop-common in the patch failed. -1 mvninstall 0m 10s hadoop-project in the patch failed. -1 mvninstall 0m 19s hadoop-tools in the patch failed. -1 mvninstall 0m 11s hadoop-azure-datalake in the patch failed. -1 mvninstall 0m 10s hadoop-tools-dist in the patch failed. +1 compile 6m 10s the patch passed with JDK v1.8.0_91 +1 javac 6m 10s the patch passed -1 compile 0m 10s root in the patch failed with JDK v1.7.0_101. -1 javac 0m 10s root in the patch failed with JDK v1.7.0_101. -1 checkstyle 1m 25s root: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) -1 mvnsite 0m 12s hadoop-common in the patch failed. -1 mvnsite 0m 12s hadoop-tools in the patch failed. -1 mvnsite 0m 12s hadoop-azure-datalake in the patch failed. -1 mvnsite 0m 10s hadoop-tools-dist in the patch failed. -1 mvneclipse 0m 9s hadoop-common in the patch failed. -1 mvneclipse 0m 24s hadoop-tools in the patch failed. -1 mvneclipse 0m 10s hadoop-tools-dist in the patch failed. -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist -1 findbugs 0m 8s hadoop-common in the patch failed. -1 findbugs 0m 11s hadoop-azure-datalake in the patch failed. -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 45s hadoop-tools in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 7s hadoop-tools-dist in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.7.0_101. -1 javadoc 0m 58s hadoop-tools in the patch failed with JDK v1.7.0_101. -1 javadoc 0m 9s hadoop-tools-dist in the patch failed with JDK v1.7.0_101. -1 unit 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_91. -1 unit 32m 26s hadoop-tools in the patch failed with JDK v1.8.0_91. +1 unit 0m 16s hadoop-azure-datalake in the patch passed with JDK v1.8.0_91. -1 unit 0m 10s hadoop-tools-dist in the patch failed with JDK v1.8.0_91. -1 unit 0m 10s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_101. -1 unit 0m 18s hadoop-tools in the patch failed with JDK v1.7.0_101. -1 unit 0m 11s hadoop-azure-datalake in the patch failed with JDK v1.7.0_101. -1 unit 0m 11s hadoop-tools-dist in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 63m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804300/HADOOP-12666-014.patch JIRA Issue HADOOP-12666 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 216c5763accf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a6e74a Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-compile-root-jdk1.7.0_101.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvnsite-hadoop-tools.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-mvneclipse-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-tools-dist.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-compile-root-jdk1.7.0_101.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure-datalake.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-tools-dist.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-common-project_hadoop-common.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-tools.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-mvneclipse-hadoop-tools_hadoop-tools-dist.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools-jdk1.7.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-tools-dist-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-tools-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9450/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Guessing that trainwreck on Jenkins was before HADOOP-13161. Will wait for an all-clear before resubmitting.

          Vishwajeet Dusane, could you review the proposed changes to CachedRefreshTokenBasedAccessTokenProvider? It does not reconfigure all running clients on setConf; it only shares an instance among clients created with the same ID. The replacement criteria were a guess based on the parameters used by ConfRefreshTokenBasedAccessTokenProvider, so please correct it as appropriate.

          HADOOP-13037 will remove the dependency on WebHDFS, largely rewriting this client. The buffering in PrivateAzureDataLakeFileSystem should also be rewritten. It's implementing something like demand-paging, but some of the control flow would be more powerful, and more understandable, if it were layered more conventionally. Configuring the client is also very complex. I tried the directions, but only arrived at a working client with Vishwajeet's help.

          The target version is 2.9, but we should hold off on backporting this before it's easier to use and maintain. I would like to commit the result of review from Chris Nauroth, Lei (Eddy) Xu, Tony Wu, Aaron Fabbri, and Sean Mackrory to trunk. It'll be easier to fixup the patch in targeted JIRAs. Committing the contract tests in HADOOP-12875 would also be helpful. This would be with the caveats from HDFS-9938: this module may be removed if it impedes WebHDFS development. Further, it should be easier to configure before we include it in a release. Is this an acceptable path forward?

          Show
          chris.douglas Chris Douglas added a comment - Guessing that trainwreck on Jenkins was before HADOOP-13161 . Will wait for an all-clear before resubmitting. Vishwajeet Dusane , could you review the proposed changes to CachedRefreshTokenBasedAccessTokenProvider ? It does not reconfigure all running clients on setConf ; it only shares an instance among clients created with the same ID. The replacement criteria were a guess based on the parameters used by ConfRefreshTokenBasedAccessTokenProvider , so please correct it as appropriate. HADOOP-13037 will remove the dependency on WebHDFS, largely rewriting this client. The buffering in PrivateAzureDataLakeFileSystem should also be rewritten. It's implementing something like demand-paging, but some of the control flow would be more powerful, and more understandable, if it were layered more conventionally. Configuring the client is also very complex. I tried the directions, but only arrived at a working client with Vishwajeet's help. The target version is 2.9, but we should hold off on backporting this before it's easier to use and maintain. I would like to commit the result of review from Chris Nauroth , Lei (Eddy) Xu , Tony Wu , Aaron Fabbri , and Sean Mackrory to trunk. It'll be easier to fixup the patch in targeted JIRAs. Committing the contract tests in HADOOP-12875 would also be helpful. This would be with the caveats from HDFS-9938 : this module may be removed if it impedes WebHDFS development. Further, it should be easier to configure before we include it in a release. Is this an acceptable path forward?
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Resubmit patch since the Hadoop-13161 is resolved.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Resubmit patch since the Hadoop-13161 is resolved.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Resubmitting patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Resubmitting patch.
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Chris Douglas - Thanks you for the change set. I reviewed the change and the change is good enough for time being. +1

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Chris Douglas - Thanks you for the change set. I reviewed the change and the change is good enough for time being. +1
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 8m 2s Maven dependency ordering for branch
          +1 mvninstall 9m 55s trunk passed
          +1 compile 10m 25s trunk passed
          +1 checkstyle 1m 44s trunk passed
          +1 mvnsite 4m 8s trunk passed
          +1 mvneclipse 1m 19s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 2m 55s trunk passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 3m 20s the patch passed
          +1 compile 9m 57s the patch passed
          +1 javac 9m 57s the patch passed
          -1 checkstyle 1m 41s root: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
          +1 mvnsite 4m 18s the patch passed
          +1 mvneclipse 2m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 11s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist
          +1 findbugs 2m 36s the patch passed
          +1 javadoc 3m 2s the patch passed
          +1 unit 0m 10s hadoop-project in the patch passed.
          +1 unit 10m 27s hadoop-common in the patch passed.
          +1 unit 0m 18s hadoop-azure-datalake in the patch passed.
          +1 unit 39m 13s hadoop-tools in the patch passed.
          +1 unit 0m 15s hadoop-tools-dist in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          120m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804431/HADOOP-12666-015.patch
          JIRA Issue HADOOP-12666
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9eca80cf7944 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4a5819d
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9466/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9466/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure-datalake hadoop-tools hadoop-tools/hadoop-tools-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9466/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 8m 2s Maven dependency ordering for branch +1 mvninstall 9m 55s trunk passed +1 compile 10m 25s trunk passed +1 checkstyle 1m 44s trunk passed +1 mvnsite 4m 8s trunk passed +1 mvneclipse 1m 19s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 1m 50s trunk passed +1 javadoc 2m 55s trunk passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 3m 20s the patch passed +1 compile 9m 57s the patch passed +1 javac 9m 57s the patch passed -1 checkstyle 1m 41s root: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) +1 mvnsite 4m 18s the patch passed +1 mvneclipse 2m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 11s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-tools/hadoop-tools-dist +1 findbugs 2m 36s the patch passed +1 javadoc 3m 2s the patch passed +1 unit 0m 10s hadoop-project in the patch passed. +1 unit 10m 27s hadoop-common in the patch passed. +1 unit 0m 18s hadoop-azure-datalake in the patch passed.