Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12535

Run FileSystem contract tests with hadoop-azure.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue proposes to implement the Hadoop FileSystem contract tests for hadoop-azure/WASB. The contract tests define the expected semantics of the FileSystem, so running these for hadoop-azure is likely to catch potential problems and improve overall quality.

      1. HADOOP-12535.001.patch
        13 kB
        madhumita chakraborty
      2. HADOOP-12535.002.patch
        20 kB
        madhumita chakraborty
      3. HADOOP-12535.003.patch
        24 kB
        madhumita chakraborty

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          +1 mvninstall 8m 0s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_66
          +1 compile 0m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_91
          +1 javac 0m 14s the patch passed
          -1 checkstyle 0m 9s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-azure (total was 23, now 23).
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91
          -1 unit 1m 15s hadoop-azure in the patch failed with JDK v1.8.0_66.
          -1 unit 1m 29s hadoop-azure in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 18s Patch generated 8 ASF License warnings.
          16m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.azure.Contract.TestAzureNativeContractCreate
            hadoop.fs.azure.Contract.TestAzureNativeContractOpen
            hadoop.fs.azure.Contract.TestAzureNativeContractRootDir
            hadoop.fs.azure.Contract.TestAzureNativeContractMkdir
            hadoop.fs.azure.Contract.TestAzureNativeContractDelete
            hadoop.fs.azure.Contract.TestAzureNativeContractRename
            hadoop.fs.azure.Contract.TestAzureNativeContractSeek
          JDK v1.7.0_91 Failed junit tests hadoop.fs.azure.Contract.TestAzureNativeContractCreate
            hadoop.fs.azure.Contract.TestAzureNativeContractOpen
            hadoop.fs.azure.Contract.TestAzureNativeContractRootDir
            hadoop.fs.azure.Contract.TestAzureNativeContractMkdir
            hadoop.fs.azure.Contract.TestAzureNativeContractDelete
            hadoop.fs.azure.Contract.TestAzureNativeContractRename
            hadoop.fs.azure.Contract.TestAzureNativeContractSeek



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782000/HADOOP-12535.001.patch
          JIRA Issue HADOOP-12535
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b0bc157ec8fa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c0537bc
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. +1 mvninstall 8m 0s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_66 +1 compile 0m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 16s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_91 +1 javac 0m 14s the patch passed -1 checkstyle 0m 9s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-azure (total was 23, now 23). +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_91 -1 unit 1m 15s hadoop-azure in the patch failed with JDK v1.8.0_66. -1 unit 1m 29s hadoop-azure in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 18s Patch generated 8 ASF License warnings. 16m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.azure.Contract.TestAzureNativeContractCreate   hadoop.fs.azure.Contract.TestAzureNativeContractOpen   hadoop.fs.azure.Contract.TestAzureNativeContractRootDir   hadoop.fs.azure.Contract.TestAzureNativeContractMkdir   hadoop.fs.azure.Contract.TestAzureNativeContractDelete   hadoop.fs.azure.Contract.TestAzureNativeContractRename   hadoop.fs.azure.Contract.TestAzureNativeContractSeek JDK v1.7.0_91 Failed junit tests hadoop.fs.azure.Contract.TestAzureNativeContractCreate   hadoop.fs.azure.Contract.TestAzureNativeContractOpen   hadoop.fs.azure.Contract.TestAzureNativeContractRootDir   hadoop.fs.azure.Contract.TestAzureNativeContractMkdir   hadoop.fs.azure.Contract.TestAzureNativeContractDelete   hadoop.fs.azure.Contract.TestAzureNativeContractRename   hadoop.fs.azure.Contract.TestAzureNativeContractSeek Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782000/HADOOP-12535.001.patch JIRA Issue HADOOP-12535 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b0bc157ec8fa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c0537bc Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8398/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          +1 mvninstall 7m 14s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_72
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_72
          +1 javac 0m 13s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_95
          +1 javac 0m 15s the patch passed
          -1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 1 new + 15 unchanged - 1 fixed = 16 total (was 16)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 1m 16s hadoop-azure in the patch passed with JDK v1.8.0_72.
          +1 unit 1m 31s hadoop-azure in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 18s Patch generated 8 ASF License warnings.
          16m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788422/HADOOP-12535.002.patch
          JIRA Issue HADOOP-12535
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux c293da4294f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2440671
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. +1 mvninstall 7m 14s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_72 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 17s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_72 +1 javac 0m 13s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_95 +1 javac 0m 15s the patch passed -1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 1 new + 15 unchanged - 1 fixed = 16 total (was 16) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 1m 16s hadoop-azure in the patch passed with JDK v1.8.0_72. +1 unit 1m 31s hadoop-azure in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 18s Patch generated 8 ASF License warnings. 16m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788422/HADOOP-12535.002.patch JIRA Issue HADOOP-12535 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c293da4294f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2440671 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8662/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          +1 mvninstall 6m 37s trunk passed
          +1 compile 0m 13s trunk passed with JDK v1.8.0_72
          +1 compile 0m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 13s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_72
          +1 javac 0m 12s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_95
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 11s the patch passed with JDK v1.7.0_95
          +1 unit 1m 9s hadoop-azure in the patch passed with JDK v1.8.0_72.
          +1 unit 1m 24s hadoop-azure in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          15m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788445/HADOOP-12535.002.patch
          JIRA Issue HADOOP-12535
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 8d4500c1c4d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2440671
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8663/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8663/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 13s trunk passed with JDK v1.8.0_72 +1 compile 0m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 13s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 16s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_72 +1 javac 0m 12s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_95 +1 javac 0m 13s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 11s the patch passed with JDK v1.7.0_95 +1 unit 1m 9s hadoop-azure in the patch passed with JDK v1.8.0_72. +1 unit 1m 24s hadoop-azure in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 15m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788445/HADOOP-12535.002.patch JIRA Issue HADOOP-12535 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8d4500c1c4d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2440671 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8663/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8663/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          madhuch-ms madhumita chakraborty added a comment -

          As we should not commit the WASB account credentials, so I disabled the tests according to https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/testing.html

          Show
          madhuch-ms madhumita chakraborty added a comment - As we should not commit the WASB account credentials, so I disabled the tests according to https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/testing.html
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi madhumita chakraborty. It's great to see these tests being added! Thank you. It looks like the contract tests have already revealed a handful of subtle bugs too.

          Here are some comments on the v002 patch.

          1. The "Contract" sub-directory should be renamed to "contract" (lower-case).
          2. In wasb.xml, I recommend using XInclude to include azure-auth-keys.xml. The definition of fs.contract.test.fs.wasb would then move to azure-auth-keys.xml. Take a look at azure-test.xml for an existing example of how to do this. azure-auth-keys.xml is a separate external file for holding a developer's Azure Storage credentials. It is ignored by revision control (listed in .gitignore at the root of the repository). This has the benefit that developers doing testing don't need to modify a file that is under revision control, and there is less risk that someone would accidentally commit private credentials to the repository.
          3. The patch includes a TestAzureNativeContractRootDir, however the tests end up getting skipped because fs.contract.test.root-tests-enabled is set to false. I tried enabling the tests, and there were some failures. I didn't dig deeper to find out if this means there is a bug or if root folder access just isn't supported fully with WASB. Could you please take a look? If it turns out that root folder access isn't fully supported, then I suggest deleting TestAzureNativeContractRootDir.
          4. HADOOP-12635 added append support for WASB, so we should be able to set fs.contract.supports-append to true and add a TestNativeAzureContractAppend suite. I tried doing this and I saw one failure in AbstractContractAppendTest#testRenameFileBeingAppended. Maybe Dushyanth could help investigate if this is a bug?
          5. It seems there ought to be a way to use the optional WASB support for atomic rename and enable fs.contract.supports-atomic-directory-delete and fs.contract.supports-atomic-rename. Have you investigated this?
          6. Let's also try enabling fs.contract.supports-getfilestatus and adding a TestAzureNativeContractGetFileStatus, which subclasses AbstractContractGetFileStatusTest.
          7. There are a few additional contract options. For the sake of completion, let's list all of these in wasb.xml: fs.contract.rename-creates-dest-dirs, fs.contract.rename-returns-false-if-dest-exists, fs.contract.rename-remove-dest-if-empty-dir, fs.contract.supports-settimes, fs.contract.supports-getfilestatus and fs.contract.supports-available-on-closed-file. The HDFS contract is at hadoop-hdfs-project/hadoop-hdfs/src/test/resources/contract/hdfs.xml. The closer we can get to matching that contract, the closer we are to a fully compatible alternative file system implementation.
          8. Let's also update the documentation at src/site/markdown/index.md with a mention of the contract tests, and update the example azure-auth-keys.xml file as I described above.
          Show
          cnauroth Chris Nauroth added a comment - Hi madhumita chakraborty . It's great to see these tests being added! Thank you. It looks like the contract tests have already revealed a handful of subtle bugs too. Here are some comments on the v002 patch. The "Contract" sub-directory should be renamed to "contract" (lower-case). In wasb.xml, I recommend using XInclude to include azure-auth-keys.xml. The definition of fs.contract.test.fs.wasb would then move to azure-auth-keys.xml. Take a look at azure-test.xml for an existing example of how to do this. azure-auth-keys.xml is a separate external file for holding a developer's Azure Storage credentials. It is ignored by revision control (listed in .gitignore at the root of the repository). This has the benefit that developers doing testing don't need to modify a file that is under revision control, and there is less risk that someone would accidentally commit private credentials to the repository. The patch includes a TestAzureNativeContractRootDir , however the tests end up getting skipped because fs.contract.test.root-tests-enabled is set to false . I tried enabling the tests, and there were some failures. I didn't dig deeper to find out if this means there is a bug or if root folder access just isn't supported fully with WASB. Could you please take a look? If it turns out that root folder access isn't fully supported, then I suggest deleting TestAzureNativeContractRootDir . HADOOP-12635 added append support for WASB, so we should be able to set fs.contract.supports-append to true and add a TestNativeAzureContractAppend suite. I tried doing this and I saw one failure in AbstractContractAppendTest#testRenameFileBeingAppended . Maybe Dushyanth could help investigate if this is a bug? It seems there ought to be a way to use the optional WASB support for atomic rename and enable fs.contract.supports-atomic-directory-delete and fs.contract.supports-atomic-rename . Have you investigated this? Let's also try enabling fs.contract.supports-getfilestatus and adding a TestAzureNativeContractGetFileStatus , which subclasses AbstractContractGetFileStatusTest . There are a few additional contract options. For the sake of completion, let's list all of these in wasb.xml: fs.contract.rename-creates-dest-dirs , fs.contract.rename-returns-false-if-dest-exists , fs.contract.rename-remove-dest-if-empty-dir , fs.contract.supports-settimes , fs.contract.supports-getfilestatus and fs.contract.supports-available-on-closed-file . The HDFS contract is at hadoop-hdfs-project/hadoop-hdfs/src/test/resources/contract/hdfs.xml. The closer we can get to matching that contract, the closer we are to a fully compatible alternative file system implementation. Let's also update the documentation at src/site/markdown/index.md with a mention of the contract tests, and update the example azure-auth-keys.xml file as I described above.
          Hide
          madhuch-ms madhumita chakraborty added a comment -

          Took care if the comments and uploaded new patch.
          Chris Nauroth Some doubts
          1. How do atomic rename and delete settings (fs.contract.supports-atomic-directory-delete and fs.contract.supports-atomic-directory-rename) get used? I dont see any reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE defined in ContractOptions. If a filesystem supports atomic rename, do they need to write new contract test?
          2. I added contract test for append. testRenameFileBeingAppended if failing. I have skipped this test for now. This test case appends a file and then renames it.

          FSDataOutputStream outputStream = getFileSystem().append(target);
          outputStream.write(dataset);
          Path renamed = new Path(testPath,”renamed”);
          Rename(target, renamed);
          outputStream.close();

          So here we see that after renaming it closes the original file stream. Was this test written in this way to maintain parity with unix file system? But WASB does not support this semantic. If the file is renamed before closing the original stream then we get 2 types of error.
          a. In append we take SelfRenewingLease on the blob. In Close we see if the lease has expired it will try to renew the lease and that will throw exception as the blob has already been deleted as part of rename.
          b. In close we commit the blocks of blockblob. So if we rename the file before closing the stream, actually the content does not get committed. And empty renamed file is created.
          cc Dushyanth

          3.Root folder access is not fully supported in WASB. So removed TestAzureNativeContractRootDir.

          Show
          madhuch-ms madhumita chakraborty added a comment - Took care if the comments and uploaded new patch. Chris Nauroth Some doubts 1. How do atomic rename and delete settings (fs.contract.supports-atomic-directory-delete and fs.contract.supports-atomic-directory-rename) get used? I dont see any reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE defined in ContractOptions. If a filesystem supports atomic rename, do they need to write new contract test? 2. I added contract test for append. testRenameFileBeingAppended if failing. I have skipped this test for now. This test case appends a file and then renames it. FSDataOutputStream outputStream = getFileSystem().append(target); outputStream.write(dataset); Path renamed = new Path(testPath,”renamed”); Rename(target, renamed); outputStream.close(); So here we see that after renaming it closes the original file stream. Was this test written in this way to maintain parity with unix file system? But WASB does not support this semantic. If the file is renamed before closing the original stream then we get 2 types of error. a. In append we take SelfRenewingLease on the blob. In Close we see if the lease has expired it will try to renew the lease and that will throw exception as the blob has already been deleted as part of rename. b. In close we commit the blocks of blockblob. So if we rename the file before closing the stream, actually the content does not get committed. And empty renamed file is created. cc Dushyanth 3.Root folder access is not fully supported in WASB. So removed TestAzureNativeContractRootDir.
          Hide
          cnauroth Chris Nauroth added a comment -

          madhumita chakraborty, thank you for the update. Patch v003 is looking good so far, and I'm running it through some final tests.

          I dont see any reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE defined in ContractOptions.

          Oops, you're right. These are unused. Perhaps these flags are just set up for documentation purposes or anticipating writing atomicity contract tests in the future. (Cc Steve Loughran.) For now, let's take this out of scope for the current patch and proceed.

          testRenameFileBeingAppended is failing.

          Yes, if this is a known deviation from HDFS semantics, then using the skip method is the right thing to do. I'd appreciate confirmation from Dushyanth that this is a difference in semantics and not in fact an unintended bug in the append implementation.

          Root folder access is not fully supported in WASB. So removed TestAzureNativeContractRootDir.

          Similar question: is this an accepted difference in semantics, or is this a bug in the handling of /? If it's not certain, then you could create a separate JIRA to track handling of /. I wouldn't want it to block committing this patch, because it's clear that we're already getting value from the tests that are working.

          Show
          cnauroth Chris Nauroth added a comment - madhumita chakraborty , thank you for the update. Patch v003 is looking good so far, and I'm running it through some final tests. I dont see any reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE defined in ContractOptions. Oops, you're right. These are unused. Perhaps these flags are just set up for documentation purposes or anticipating writing atomicity contract tests in the future. (Cc Steve Loughran .) For now, let's take this out of scope for the current patch and proceed. testRenameFileBeingAppended is failing. Yes, if this is a known deviation from HDFS semantics, then using the skip method is the right thing to do. I'd appreciate confirmation from Dushyanth that this is a difference in semantics and not in fact an unintended bug in the append implementation. Root folder access is not fully supported in WASB. So removed TestAzureNativeContractRootDir. Similar question: is this an accepted difference in semantics, or is this a bug in the handling of /? If it's not certain, then you could create a separate JIRA to track handling of /. I wouldn't want it to block committing this patch, because it's clear that we're already getting value from the tests that are working.
          Hide
          dchickabasapa Dushyanth added a comment -

          madhumita chakraborty This is a known deviation for append support in WASB. Append expects the application to enforce single modifier semantics for a file. Unlike HDFS where we have lease or lock on the path, WASB does not have the same semantics and we expect the applications to enforce that the file is not modified (rename or delete) during append.

          I think we can skip this test for Append at this time and re-enable it when we bring append support to have the same semantics as HDFS.

          cc Chris Nauroth

          Show
          dchickabasapa Dushyanth added a comment - madhumita chakraborty This is a known deviation for append support in WASB. Append expects the application to enforce single modifier semantics for a file. Unlike HDFS where we have lease or lock on the path, WASB does not have the same semantics and we expect the applications to enforce that the file is not modified (rename or delete) during append. I think we can skip this test for Append at this time and re-enable it when we bring append support to have the same semantics as HDFS. cc Chris Nauroth
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_72
          +1 compile 0m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_72
          +1 javac 0m 11s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_95
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16)
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_72.
          +1 unit 1m 27s hadoop-azure in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          15m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789273/HADOOP-12535.003.patch
          JIRA Issue HADOOP-12535
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2da2e67bba39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4343a4c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8701/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8701/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_72 +1 compile 0m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 15s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_72 +1 javac 0m 11s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_95 +1 javac 0m 13s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: patch generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16) +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_72. +1 unit 1m 27s hadoop-azure in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 15m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789273/HADOOP-12535.003.patch JIRA Issue HADOOP-12535 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2da2e67bba39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4343a4c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8701/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8701/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I have committed this to trunk, branch-2 and branch-2.8. I have filed HADOOP-12836 for follow-up on trying to run the root directory tests against hadoop-azure.

          madhumita chakraborty, thank you for contributing this patch. It's great to see these tests executing with hadoop-azure, and we can see that it's already been helpful for flagging a few subtle semantic issues.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I have committed this to trunk, branch-2 and branch-2.8. I have filed HADOOP-12836 for follow-up on trying to run the root directory tests against hadoop-azure. madhumita chakraborty , thank you for contributing this patch. It's great to see these tests executing with hadoop-azure, and we can see that it's already been helpful for flagging a few subtle semantic issues.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9359 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9359/)
          HADOOP-12535. Run FileSystem contract tests with hadoop-azure. (cnauroth: rev 3369a4f6916f12e9d6b97072badd1b176be443bd)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/NativeAzureFileSystemContract.java
          • hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractAppend.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractDelete.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractOpen.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractGetFileStatus.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractRename.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractSeek.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractCreate.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractMkdir.java
          • hadoop-tools/hadoop-azure/src/test/resources/wasb.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9359 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9359/ ) HADOOP-12535 . Run FileSystem contract tests with hadoop-azure. (cnauroth: rev 3369a4f6916f12e9d6b97072badd1b176be443bd) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/NativeAzureFileSystemContract.java hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractAppend.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractDelete.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractOpen.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractGetFileStatus.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractRename.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractSeek.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractCreate.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractMkdir.java hadoop-tools/hadoop-azure/src/test/resources/wasb.xml hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              madhuch-ms madhumita chakraborty
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development