Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: tools
    • Labels:
    • Target Version/s:

      Description

      The regression fixed by HADOOP-12689 had no unit tests to expose the problem. Add filesystem tests according to http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/testing.html for the s3 scheme.

      1. HADOOP-12696.01.patch
        21 kB
        Matthew Paduano
      2. HADOOP-12696.02.patch
        29 kB
        Matthew Paduano
      3. log.fail2
        107 kB
        Matthew Paduano
      4. log.hdfs
        19 kB
        Matthew Paduano
      5. log.patch02
        3 kB
        Matthew Paduano
      6. log.s3a
        19 kB
        Matthew Paduano
      7. log.s3n
        16 kB
        Matthew Paduano
      8. log.succ
        3 kB
        Matthew Paduano

        Issue Links

          Activity

          Hide
          mattpaduano Matthew Paduano added a comment -

          test log with regression from HADOOP-12689 being caught

          Show
          mattpaduano Matthew Paduano added a comment - test log with regression from HADOOP-12689 being caught
          Hide
          mattpaduano Matthew Paduano added a comment -

          test log against current trunk - 47 tests run, 10 skipped, 37 passed

          Show
          mattpaduano Matthew Paduano added a comment - test log against current trunk - 47 tests run, 10 skipped, 37 passed
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          +1 mvninstall 7m 59s trunk passed
          +1 compile 8m 46s trunk passed with JDK v1.8.0_66
          +1 compile 8m 56s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 58s trunk passed
          +1 mvnsite 1m 19s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 14s trunk passed
          +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 50s the patch passed
          +1 compile 7m 56s the patch passed with JDK v1.8.0_66
          +1 javac 7m 56s the patch passed
          +1 compile 8m 46s the patch passed with JDK v1.7.0_91
          +1 javac 8m 46s the patch passed
          +1 checkstyle 0m 59s the patch passed
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91
          +1 unit 7m 11s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66.
          -1 unit 6m 54s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          75m 41s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.fs.TestLocalFsFCStatistics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781144/HADOOP-12696.01.patch
          JIRA Issue HADOOP-12696
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
          uname Linux 4294f69c6180 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89022f8
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 8m 46s trunk passed with JDK v1.8.0_66 +1 compile 8m 56s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 58s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 14s trunk passed +1 javadoc 1m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 50s the patch passed +1 compile 7m 56s the patch passed with JDK v1.8.0_66 +1 javac 7m 56s the patch passed +1 compile 8m 46s the patch passed with JDK v1.7.0_91 +1 javac 8m 46s the patch passed +1 checkstyle 0m 59s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 32s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91 +1 unit 7m 11s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66. -1 unit 6m 54s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 75m 41s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.fs.TestLocalFsFCStatistics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781144/HADOOP-12696.01.patch JIRA Issue HADOOP-12696 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux 4294f69c6180 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89022f8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8363/console This message was automatically generated.
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Matt!
          Thanks for the patch. You seem to have copied s3.xml from hdfs.xml. I wonder if s3n.xml is not a better XML to copy. e.g. is-blobstore should probably be true for S3 too, shouldn't it?

          Show
          raviprak Ravi Prakash added a comment - Hi Matt! Thanks for the patch. You seem to have copied s3.xml from hdfs.xml. I wonder if s3n.xml is not a better XML to copy. e.g. is-blobstore should probably be true for S3 too, shouldn't it?
          Hide
          mattpaduano Matthew Paduano added a comment -

          to copy.

          I dunno. maybe. maybe not. I did see ContractOptions.IS_BLOBSTORE
          in a
          few places in the test code when I was reviewing the failures. But it
          didn't seem
          like it would have fixed the test issues to me whichever way it pointed.

          But, it is easy enough to run the tests with various contracts. I
          removed
          all the forced skip @overrides from the test code and re-ran the tests
          with
          the original s3.xml (== hdfs.xml) and also with s3a.xml and s3n.xml.
          The
          same 10 tests still have issues, but the issues are different

          I attached the output from all three test runs to this email. summary:

          hdfs Tests run: 47, Failures: 3, Errors: 7, Skipped: 0
          s3n Tests run: 47, Failures: 2, Errors: 6, Skipped: 2
          s3a Tests run: 47, Failures: 3, Errors: 7, Skipped: 2

          the s3n case does seem to account for two of the errors and
          instead they are "auto skipped". testCreatedFileIsImmediatelyVisible
          (which is the atomic file option I think) and
          testOverwriteNonEmptyDirectory
          (not sure of the option). I am not sure I understand the skip code...
          it is sort of like a intermediate state between fail and pass...

          matt

          Show
          mattpaduano Matthew Paduano added a comment - to copy. I dunno. maybe. maybe not. I did see ContractOptions.IS_BLOBSTORE in a few places in the test code when I was reviewing the failures. But it didn't seem like it would have fixed the test issues to me whichever way it pointed. But, it is easy enough to run the tests with various contracts. I removed all the forced skip @overrides from the test code and re-ran the tests with the original s3.xml (== hdfs.xml) and also with s3a.xml and s3n.xml. The same 10 tests still have issues, but the issues are different I attached the output from all three test runs to this email. summary: hdfs Tests run: 47, Failures: 3, Errors: 7, Skipped: 0 s3n Tests run: 47, Failures: 2, Errors: 6, Skipped: 2 s3a Tests run: 47, Failures: 3, Errors: 7, Skipped: 2 the s3n case does seem to account for two of the errors and instead they are "auto skipped". testCreatedFileIsImmediatelyVisible (which is the atomic file option I think) and testOverwriteNonEmptyDirectory (not sure of the option). I am not sure I understand the skip code... it is sort of like a intermediate state between fail and pass... matt
          Hide
          mattpaduano Matthew Paduano added a comment -

          oops... I did not intend to reply to that address :/

          but since I did, I should also mention that I think those 10
          issues can be fixed in one of S3FileSystem, S3InputStream
          and/or the contract and AbstractFSContractTestBase.

          Show
          mattpaduano Matthew Paduano added a comment - oops... I did not intend to reply to that address :/ but since I did, I should also mention that I think those 10 issues can be fixed in one of S3FileSystem, S3InputStream and/or the contract and AbstractFSContractTestBase.
          Hide
          mattpaduano Matthew Paduano added a comment -

          log.patch02 is the output of running:

          mvn -l /tmp/log test -Ptests-on -Dtest=org.apache.hadoop.fs.contract.TestS3Contract*

          Show
          mattpaduano Matthew Paduano added a comment - log.patch02 is the output of running: mvn -l /tmp/log test -Ptests-on -Dtest=org.apache.hadoop.fs.contract.TestS3Contract*
          Hide
          mattpaduano Matthew Paduano added a comment -

          patch02 contains fixes for S3FileSystem, S3InputStream and one additional ContractOption with corresponding code in the SeekTest.

          Tests run: 47, Failures: 0, Errors: 0, Skipped: 1

          Show
          mattpaduano Matthew Paduano added a comment - patch02 contains fixes for S3FileSystem, S3InputStream and one additional ContractOption with corresponding code in the SeekTest. Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 11 new or modified test files.
          +1 mvninstall 8m 45s trunk passed
          +1 compile 9m 27s trunk passed with JDK v1.8.0_66
          +1 compile 10m 3s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 2m 39s trunk passed
          +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 26s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 59s the patch passed
          +1 compile 8m 23s the patch passed with JDK v1.8.0_66
          +1 javac 8m 23s the patch passed
          +1 compile 8m 51s the patch passed with JDK v1.7.0_91
          +1 javac 8m 51s the patch passed
          -1 checkstyle 0m 59s Patch generated 3 new checkstyle issues in root (total was 52, now 52).
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 33s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 19s the patch passed with JDK v1.7.0_91
          -1 unit 7m 24s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          82m 4s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.TestFsShellReturnCode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781367/HADOOP-12696.02.patch
          JIRA Issue HADOOP-12696
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
          uname Linux c5ca694b3755 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 109e528
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 11 new or modified test files. +1 mvninstall 8m 45s trunk passed +1 compile 9m 27s trunk passed with JDK v1.8.0_66 +1 compile 10m 3s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 9s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 59s the patch passed +1 compile 8m 23s the patch passed with JDK v1.8.0_66 +1 javac 8m 23s the patch passed +1 compile 8m 51s the patch passed with JDK v1.7.0_91 +1 javac 8m 51s the patch passed -1 checkstyle 0m 59s Patch generated 3 new checkstyle issues in root (total was 52, now 52). +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 33s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 19s the patch passed with JDK v1.7.0_91 -1 unit 7m 24s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66. +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 82m 4s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.TestFsShellReturnCode Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781367/HADOOP-12696.02.patch JIRA Issue HADOOP-12696 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux c5ca694b3755 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 109e528 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8372/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Matthew Paduano, please only use Target-version for expressing your intention. Committers set the fix-version at commit time. FYI.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Matthew Paduano , please only use Target-version for expressing your intention. Committers set the fix-version at commit time. FYI.
          Hide
          raviprak Ravi Prakash added a comment -

          The changes look good to me. We are changing Exception types to subclasses in some cases so we should be fine. I'll commit Monday evening if there are no objections.

          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.046 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractDelete
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.904 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractSeek
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.517 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractRename
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 34.526 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractCreate
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.248 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractOpen
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.245 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractMkdir
          Running org.apache.hadoop.fs.contract.s3.TestS3ContractRootDir
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.864 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractRootDir
          
          Results :
          
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
          
          Show
          raviprak Ravi Prakash added a comment - The changes look good to me. We are changing Exception types to subclasses in some cases so we should be fine. I'll commit Monday evening if there are no objections. ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.contract.s3.TestS3ContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.046 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractDelete Running org.apache.hadoop.fs.contract.s3.TestS3ContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.904 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractSeek Running org.apache.hadoop.fs.contract.s3.TestS3ContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.517 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractRename Running org.apache.hadoop.fs.contract.s3.TestS3ContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 34.526 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractCreate Running org.apache.hadoop.fs.contract.s3.TestS3ContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.248 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractOpen Running org.apache.hadoop.fs.contract.s3.TestS3ContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.245 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractMkdir Running org.apache.hadoop.fs.contract.s3.TestS3ContractRootDir Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.864 sec - in org.apache.hadoop.fs.contract.s3.TestS3ContractRootDir Results : Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
          Hide
          raviprak Ravi Prakash added a comment -

          Also, with regards to the 1 skipped test, I found this helpful message in the logs:

          2016-01-15 16:18:42,282 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible
          
          Show
          raviprak Ravi Prakash added a comment - Also, with regards to the 1 skipped test, I found this helpful message in the logs: 2016-01-15 16:18:42,282 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible
          Hide
          raviprak Ravi Prakash added a comment -

          Also for s3n

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.296 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.855 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          vi Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.537 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.402 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 43.637 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.472 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.469 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          
          Results :
          
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 3
          

          The skipped tests

          2016-01-15 16:29:49,262 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Object store allows a file to overwrite a directory
          2016-01-15 16:29:55,553 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible
          2016-01-15 16:29:58,982 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: blobstores can't distinguish empty directories from files
          

          For s3a

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.099 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.163 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.361 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.973 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.859 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.267 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 30.839 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          
          Results :
          
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 3
          

          The skipped tests

          org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:57:59,559 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Object store allows a file to overwrite a directory
          org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:58:03,127 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible
          org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:58:06,419 INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: blobstores can't distinguish empty directories from files
          
          Show
          raviprak Ravi Prakash added a comment - Also for s3n ------------------------------------------------------- T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.296 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.855 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir vi Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.537 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.402 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 43.637 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.472 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.469 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Results : Tests run: 47, Failures: 0, Errors: 0, Skipped: 3 The skipped tests 2016-01-15 16:29:49,262 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Object store allows a file to overwrite a directory 2016-01-15 16:29:55,553 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible 2016-01-15 16:29:58,982 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: blobstores can't distinguish empty directories from files For s3a ------------------------------------------------------- T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.099 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.163 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.361 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.973 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.859 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.267 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 30.839 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Results : Tests run: 47, Failures: 0, Errors: 0, Skipped: 3 The skipped tests org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:57:59,559 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Object store allows a file to overwrite a directory org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:58:03,127 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: Filesystem is an object store and newly created files are not immediately visible org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate-output.txt:2016-01-18 10:58:06,419 INFO contract.ContractTestUtils (ContractTestUtils.java:skip(424)) - Skipping: blobstores can't distinguish empty directories from files
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks for the contribution Matt. I've committed this patch to trunk, branch-2 and branch-2.8.

          Show
          raviprak Ravi Prakash added a comment - Thanks for the contribution Matt. I've committed this patch to trunk, branch-2 and branch-2.8.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9138/)
          HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt (raviprak: rev 1acc509b45d58c0eb7e83ea1ba13169410be0dbe)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractSeek.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/S3Contract.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractRename.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractCreate.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractMkdir.java
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3InputStream.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractOpen.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractOptions.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractRootDir.java
          • hadoop-tools/hadoop-aws/src/test/resources/contract/s3.xml
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractDelete.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9138/ ) HADOOP-12696 . Add tests for S3FileSystem Contract. Contributed by Matt (raviprak: rev 1acc509b45d58c0eb7e83ea1ba13169410be0dbe) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractSeek.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/S3Contract.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractRename.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractCreate.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractMkdir.java hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3InputStream.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractOpen.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractOptions.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractRootDir.java hadoop-tools/hadoop-aws/src/test/resources/contract/s3.xml hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3/TestS3ContractDelete.java
          Hide
          cnauroth Chris Nauroth added a comment -

          TestS3ContractRootDir has some failures when running on branch-2. The tests pass on trunk. I filed HADOOP-12815 for follow-up. Could one of the original contributors here please take a look? Thank you.

          Show
          cnauroth Chris Nauroth added a comment - TestS3ContractRootDir has some failures when running on branch-2. The tests pass on trunk. I filed HADOOP-12815 for follow-up. Could one of the original contributors here please take a look? Thank you.

            People

            • Assignee:
              mattpaduano Matthew Paduano
              Reporter:
              mattpaduano Matthew Paduano
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development