Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13502

Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The fs.contract.is-blobstore flag guards against execution of several contract tests to account for known limitations with blob stores. However, the name is not entirely accurate, because it's still possible that a file system implemented against a blob store could pass those tests, depending on whether or not the implementation matches the semantics of HDFS. This issue proposes to rename the flag or split it into different flags with different definitions for the semantics covered by the current flag.

      1. HADOOP-13502-branch-2.001.patch
        7 kB
        Chris Nauroth
      2. HADOOP-13502-branch-2.002.patch
        7 kB
        Chris Nauroth
      3. HADOOP-13502-branch-2.003.patch
        7 kB
        Chris Nauroth
      4. HADOOP-13502-branch-2.004.patch
        8 kB
        Chris Nauroth
      5. HADOOP-13502-trunk.004.patch
        7 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10688 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10688/)
          HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive (cnauroth: rev 1f8490a5bacd98d0d371447ada3b31f93ca40a4e)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
          • (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3a.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/resources/contract/ftp.xml
          • (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3n.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractOptions.java
          • (edit) .gitignore
          • (edit) hadoop-tools/hadoop-openstack/src/test/resources/contract/swift.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10688 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10688/ ) HADOOP-13502 . Split fs.contract.is-blobstore flag into more descriptive (cnauroth: rev 1f8490a5bacd98d0d371447ada3b31f93ca40a4e) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3a.xml (edit) hadoop-common-project/hadoop-common/src/test/resources/contract/ftp.xml (edit) hadoop-tools/hadoop-aws/src/test/resources/contract/s3n.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractOptions.java (edit) .gitignore (edit) hadoop-tools/hadoop-openstack/src/test/resources/contract/swift.xml
          Hide
          cnauroth Chris Nauroth added a comment -

          Xiaoyu, thank you for your code review too.

          Show
          cnauroth Chris Nauroth added a comment - Xiaoyu, thank you for your code review too.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve, thank you for your review. I committed this to trunk, branch-2 and branch-2.8 after completing another test run.

          Show
          cnauroth Chris Nauroth added a comment - Steve, thank you for your review. I committed this to trunk, branch-2 and branch-2.8 after completing another test run.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          LGTM.

          +1 —if you do a runthrough applied to the latest code before you check it in

          Show
          stevel@apache.org Steve Loughran added a comment - LGTM. +1 —if you do a runthrough applied to the latest code before you check it in
          Hide
          cnauroth Chris Nauroth added a comment -

          The warnings flagged by pre-commit for the trunk patch are not relevant.

          Show
          cnauroth Chris Nauroth added a comment - The warnings flagged by pre-commit for the trunk patch are not relevant.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 35s trunk passed
          +1 compile 8m 8s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 10m 31s trunk passed
          +1 mvneclipse 1m 30s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: .
          +1 findbugs 2m 47s trunk passed
          +1 javadoc 5m 6s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 9m 33s the patch passed
          +1 compile 8m 21s the patch passed
          +1 javac 8m 21s the patch passed
          +1 checkstyle 1m 39s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20)
          +1 mvnsite 10m 5s the patch passed
          +1 mvneclipse 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: .
          +1 findbugs 2m 44s the patch passed
          +1 javadoc 4m 22s the patch passed
          -1 unit 101m 0s root in the patch failed.
          -1 asflicense 0m 29s The patch generated 2 ASF License warnings.
          199m 54s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
            hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832572/HADOOP-13502-trunk.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux bc4d72ba58d4 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 96b1266
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 35s trunk passed +1 compile 8m 8s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 10m 31s trunk passed +1 mvneclipse 1m 30s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 47s trunk passed +1 javadoc 5m 6s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 9m 33s the patch passed +1 compile 8m 21s the patch passed +1 javac 8m 21s the patch passed +1 checkstyle 1m 39s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) +1 mvnsite 10m 5s the patch passed +1 mvneclipse 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 44s the patch passed +1 javadoc 4m 22s the patch passed -1 unit 101m 0s root in the patch failed. -1 asflicense 0m 29s The patch generated 2 ASF License warnings. 199m 54s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832572/HADOOP-13502-trunk.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux bc4d72ba58d4 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96b1266 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10724/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve Loughran, have you had a chance to try the hadoop-openstack tests again with this patch? I commented a few days ago saying that I couldn't repro the failure that you saw.

          Also, I am now attaching a separate patch file for trunk. The only difference is the omission of s3.xml, which does not exist on trunk.

          Show
          cnauroth Chris Nauroth added a comment - Steve Loughran , have you had a chance to try the hadoop-openstack tests again with this patch? I commented a few days ago saying that I couldn't repro the failure that you saw. Also, I am now attaching a separate patch file for trunk. The only difference is the omission of s3.xml, which does not exist on trunk.
          Hide
          cnauroth Chris Nauroth added a comment -

          The warnings from the last pre-commit run are not relevant to the patch.

          Show
          cnauroth Chris Nauroth added a comment - The warnings from the last pre-commit run are not relevant to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          0 mvndep 0m 59s Maven dependency ordering for branch
          +1 mvninstall 7m 3s branch-2 passed
          +1 compile 6m 34s branch-2 passed with JDK v1.8.0_101
          +1 compile 7m 13s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 30s branch-2 passed
          +1 mvnsite 9m 42s branch-2 passed
          +1 mvneclipse 0m 56s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: .
          +1 findbugs 2m 52s branch-2 passed
          +1 javadoc 5m 2s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 6m 34s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 8m 13s the patch passed
          +1 compile 5m 37s the patch passed with JDK v1.8.0_101
          +1 javac 5m 37s the patch passed
          +1 compile 6m 53s the patch passed with JDK v1.7.0_111
          +1 javac 6m 53s the patch passed
          +1 checkstyle 1m 30s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20)
          +1 mvnsite 9m 53s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: .
          +1 findbugs 3m 46s the patch passed
          +1 javadoc 5m 27s the patch passed with JDK v1.8.0_101
          +1 javadoc 6m 43s the patch passed with JDK v1.7.0_111
          -1 unit 13m 28s root in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 28s The patch generated 2 ASF License warnings.
          149m 28s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.net.TestDNS
          JDK v1.7.0_111 Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831788/HADOOP-13502-branch-2.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux d4a07d028e05 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 6ed335a
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/patch-unit-root-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 0m 59s Maven dependency ordering for branch +1 mvninstall 7m 3s branch-2 passed +1 compile 6m 34s branch-2 passed with JDK v1.8.0_101 +1 compile 7m 13s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 9m 42s branch-2 passed +1 mvneclipse 0m 56s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 52s branch-2 passed +1 javadoc 5m 2s branch-2 passed with JDK v1.8.0_101 +1 javadoc 6m 34s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 8m 13s the patch passed +1 compile 5m 37s the patch passed with JDK v1.8.0_101 +1 javac 5m 37s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_111 +1 javac 6m 53s the patch passed +1 checkstyle 1m 30s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) +1 mvnsite 9m 53s the patch passed +1 mvneclipse 0m 53s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 3m 46s the patch passed +1 javadoc 5m 27s the patch passed with JDK v1.8.0_101 +1 javadoc 6m 43s the patch passed with JDK v1.7.0_111 -1 unit 13m 28s root in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 28s The patch generated 2 ASF License warnings. 149m 28s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.net.TestDNS JDK v1.7.0_111 Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831788/HADOOP-13502-branch-2.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d4a07d028e05 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 6ed335a Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/patch-unit-root-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10676/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch 004. The only difference is that I'm taking this opportunity to update .gitignore for the hadoop-openstack auth-keys.xml file.

          Steve, I cannot repro the hadoop-openstack test failures that you mentioned. TestSwiftContractCreate runs clean, and I see the following in the log showing that the contract XML flags are causing tests to be skipped correctly.

          2016-10-05 10:30:31,296 [Thread-0] INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: This Filesystem allows a file to overwrite a directory
          
          2016-10-05 10:30:35,037 [Thread-5] INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: This Filesystem delays visibility of newly created files
          
          2016-10-05 10:30:37,113 [Thread-6] INFO  contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: blobstores can't distinguish empty directories from files
          

          Could you please try testing again? Maybe you need a fresh "mvn install" to get the hadoop-common changes installed.

          This is my first time running the hadoop-openstack tests. I'm consistently getting failures due to socket read timeouts in these tests:

          • TestFSMainOperationsSwift#testGlobStatusFilterWithSomePathMatchesAndTrivialFilter
          • TestFSMainOperationsSwift#testWriteReadAndDeleteTwoBlocks
          • TestFSMainOperationsSwift#testRenameFileToExistingParent
          • TestSwiftFileSystemRename#testRenameFile
          • TestSwiftFileSystemRename#testRenameFileIntoExistingDirectory

          I'm running against the public Rackspace DFW endpoint, not a VM within Rackspace, so I assume public Internet latency is a contributing factor. I did full test runs before and after my patch, and I confirmed that I was getting the same results before and after.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch 004. The only difference is that I'm taking this opportunity to update .gitignore for the hadoop-openstack auth-keys.xml file. Steve, I cannot repro the hadoop-openstack test failures that you mentioned. TestSwiftContractCreate runs clean, and I see the following in the log showing that the contract XML flags are causing tests to be skipped correctly. 2016-10-05 10:30:31,296 [ Thread -0] INFO contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: This Filesystem allows a file to overwrite a directory 2016-10-05 10:30:35,037 [ Thread -5] INFO contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: This Filesystem delays visibility of newly created files 2016-10-05 10:30:37,113 [ Thread -6] INFO contract.ContractTestUtils (ContractTestUtils.java:skip(432)) - Skipping: blobstores can't distinguish empty directories from files Could you please try testing again? Maybe you need a fresh "mvn install" to get the hadoop-common changes installed. This is my first time running the hadoop-openstack tests. I'm consistently getting failures due to socket read timeouts in these tests: TestFSMainOperationsSwift#testGlobStatusFilterWithSomePathMatchesAndTrivialFilter TestFSMainOperationsSwift#testWriteReadAndDeleteTwoBlocks TestFSMainOperationsSwift#testRenameFileToExistingParent TestSwiftFileSystemRename#testRenameFile TestSwiftFileSystemRename#testRenameFileIntoExistingDirectory I'm running against the public Rackspace DFW endpoint, not a VM within Rackspace, so I assume public Internet latency is a contributing factor. I did full test runs before and after my patch, and I confirmed that I was getting the same results before and after.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          openstack tests are failing against rackspace US

           T E S T S
          -------------------------------------------------------
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate
          Tests run: 6, Failures: 1, Errors: 1, Skipped: 1, Time elapsed: 49.79 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate
          testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate)  Time elapsed: 14.033 sec  <<< FAILURE!
          java.lang.AssertionError: write of file over dir succeeded
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testOverwriteNonEmptyDirectory(AbstractContractCreateTest.java:145)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          testCreatedFileIsImmediatelyVisible(org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate)  Time elapsed: 7.317 sec  <<< ERROR!
          java.io.FileNotFoundException: expected path to be visible before anything written: not found swift://jenkins.rackspace/test/testCreatedFileIsImmediatelyVisible in swift://jenkins.rackspace/test
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:698)
          	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
          	at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsImmediatelyVisible(AbstractContractCreateTest.java:178)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.fs.swift.contract.TestSwiftContractDelete
          

          it's a bit confusing as I'm also getting timeouts on other operations; I don't think I'm getting a clean run

          Show
          stevel@apache.org Steve Loughran added a comment - openstack tests are failing against rackspace US T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate Tests run: 6, Failures: 1, Errors: 1, Skipped: 1, Time elapsed: 49.79 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate) Time elapsed: 14.033 sec <<< FAILURE! java.lang.AssertionError: write of file over dir succeeded at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testOverwriteNonEmptyDirectory(AbstractContractCreateTest.java:145) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) testCreatedFileIsImmediatelyVisible(org.apache.hadoop.fs.swift.contract.TestSwiftContractCreate) Time elapsed: 7.317 sec <<< ERROR! java.io.FileNotFoundException: expected path to be visible before anything written: not found swift: //jenkins.rackspace/test/testCreatedFileIsImmediatelyVisible in swift://jenkins.rackspace/test at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:698) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294) at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsImmediatelyVisible(AbstractContractCreateTest.java:178) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.swift.contract.TestSwiftContractDelete it's a bit confusing as I'm also getting timeouts on other operations; I don't think I'm getting a clean run
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          0 mvndep 1m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 21s branch-2 passed
          +1 compile 6m 55s branch-2 passed with JDK v1.8.0_101
          +1 compile 7m 15s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 30s branch-2 passed
          +1 mvnsite 1m 43s branch-2 passed
          +1 mvneclipse 0m 46s branch-2 passed
          +1 findbugs 3m 7s branch-2 passed
          +1 javadoc 1m 29s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 36s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 6m 8s the patch passed with JDK v1.8.0_101
          +1 javac 6m 8s the patch passed
          +1 compile 7m 9s the patch passed with JDK v1.7.0_111
          +1 javac 7m 9s the patch passed
          +1 checkstyle 1m 35s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 findbugs 4m 2s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_111
          -1 unit 7m 45s hadoop-common in the patch failed with JDK v1.7.0_111.
          +1 unit 0m 22s hadoop-openstack in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 29s hadoop-aws in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          103m 22s



          Reason Tests
          JDK v1.7.0_111 Failed junit tests hadoop.ipc.TestRPCWaitForProxy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831611/HADOOP-13502-branch-2.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 34a42506fcbe 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 213f0ad
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 1m 15s Maven dependency ordering for branch +1 mvninstall 7m 21s branch-2 passed +1 compile 6m 55s branch-2 passed with JDK v1.8.0_101 +1 compile 7m 15s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 1m 43s branch-2 passed +1 mvneclipse 0m 46s branch-2 passed +1 findbugs 3m 7s branch-2 passed +1 javadoc 1m 29s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 36s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 6m 8s the patch passed with JDK v1.8.0_101 +1 javac 6m 8s the patch passed +1 compile 7m 9s the patch passed with JDK v1.7.0_111 +1 javac 7m 9s the patch passed +1 checkstyle 1m 35s root: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 55s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 3s The patch has no ill-formed XML file. +1 findbugs 4m 2s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_111 -1 unit 7m 45s hadoop-common in the patch failed with JDK v1.7.0_111. +1 unit 0m 22s hadoop-openstack in the patch passed with JDK v1.7.0_111. +1 unit 0m 29s hadoop-aws in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 103m 22s Reason Tests JDK v1.7.0_111 Failed junit tests hadoop.ipc.TestRPCWaitForProxy Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831611/HADOOP-13502-branch-2.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 34a42506fcbe 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 213f0ad Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10664/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Also, the patch still needs a test run for hadoop-openstack.

          Show
          cnauroth Chris Nauroth added a comment - Also, the patch still needs a test run for hadoop-openstack.
          Hide
          cnauroth Chris Nauroth added a comment -

          The whitespace check triggered warnings on a file I didn't change in this patch, so it's not relevant.

          Xiaoyu, thank you for your review. I'll hold the commit in case Steve has additional feedback.

          Show
          cnauroth Chris Nauroth added a comment - The whitespace check triggered warnings on a file I didn't change in this patch, so it's not relevant. Xiaoyu, thank you for your review. I'll hold the commit in case Steve has additional feedback.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          0 mvndep 1m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 43s branch-2 passed
          +1 compile 5m 35s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 41s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 26s branch-2 passed
          +1 mvnsite 1m 43s branch-2 passed
          +1 mvneclipse 0m 48s branch-2 passed
          +1 findbugs 2m 51s branch-2 passed
          +1 javadoc 1m 18s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 33s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 6m 13s the patch passed with JDK v1.8.0_101
          +1 javac 6m 13s the patch passed
          +1 compile 7m 35s the patch passed with JDK v1.7.0_111
          +1 javac 7m 35s the patch passed
          +1 checkstyle 1m 44s the patch passed
          +1 mvnsite 2m 3s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 13s the patch passed
          +1 javadoc 1m 32s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 50s the patch passed with JDK v1.7.0_111
          +1 unit 8m 55s hadoop-common in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 29s hadoop-aws in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          103m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831601/HADOOP-13502-branch-2.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 26c445425510 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 213f0ad
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 1m 14s Maven dependency ordering for branch +1 mvninstall 6m 43s branch-2 passed +1 compile 5m 35s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 41s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 26s branch-2 passed +1 mvnsite 1m 43s branch-2 passed +1 mvneclipse 0m 48s branch-2 passed +1 findbugs 2m 51s branch-2 passed +1 javadoc 1m 18s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 33s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 6m 13s the patch passed with JDK v1.8.0_101 +1 javac 6m 13s the patch passed +1 compile 7m 35s the patch passed with JDK v1.7.0_111 +1 javac 7m 35s the patch passed +1 checkstyle 1m 44s the patch passed +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 55s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 13s the patch passed +1 javadoc 1m 32s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_111 +1 unit 8m 55s hadoop-common in the patch passed with JDK v1.7.0_111. +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_111. +1 unit 0m 29s hadoop-aws in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 103m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831601/HADOOP-13502-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 26c445425510 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 213f0ad Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10660/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          +1 patch 003 pending Jenkins.

          Show
          xyao Xiaoyu Yao added a comment - +1 patch 003 pending Jenkins.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you, Xiaoyu. That's a good suggestion. Here is patch 003.

          Show
          cnauroth Chris Nauroth added a comment - Thank you, Xiaoyu. That's a good suggestion. Here is patch 003.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Chris Nauroth for the update. Just one NIT:
          in AbstractContractCreateTest.java, can we remove the assumption of object store in the comments and the skip message to match the description of the new flag, which should be applicable to all HCFS?

           // object store: downgrade to a skip so that the failure is visible
           // in test results
           skip("Filesystem is an object store and newly created files are not immediately visible");
          
          Show
          xyao Xiaoyu Yao added a comment - Thanks Chris Nauroth for the update. Just one NIT: in AbstractContractCreateTest.java, can we remove the assumption of object store in the comments and the skip message to match the description of the new flag, which should be applicable to all HCFS? // object store: downgrade to a skip so that the failure is visible // in test results skip( "Filesystem is an object store and newly created files are not immediately visible" );
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve and I spoke offline. I had a misunderstanding about his last comment. His point was that we really don't have any other known object stores. There are other file system implementations, but as far as we know, they are not object stores, and therefore they would never set the is-blobstore flag.

          Based on that, I'm uploading patch 002. This retains is-blobstore as an informational flag. It is not deprecated. The tests no longer need to check for is-blobstore, because all known object stores have had their contract XML options updated. As a side effect, the patch no longer needs to touch wasb.xml.

          Show
          cnauroth Chris Nauroth added a comment - Steve and I spoke offline. I had a misunderstanding about his last comment. His point was that we really don't have any other known object stores. There are other file system implementations, but as far as we know, they are not object stores, and therefore they would never set the is-blobstore flag. Based on that, I'm uploading patch 002. This retains is-blobstore as an informational flag. It is not deprecated. The tests no longer need to check for is-blobstore, because all known object stores have had their contract XML options updated. As a side effect, the patch no longer needs to touch wasb.xml.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm aware of closed-source implementations that may be using the HCFS tests.

          Really this comes down to compatibility policy for the HCFS tests. I don't think we've discussed that in the past, and there are currently no audience/stability annotations. If we are committed to backward compatibility, then we're better off with the approach I took in patch 001. If we are not committed to backward compatibility for the HCFS tests, then we're free to change anything we want and repurpose is-blobstore as a purely descriptive flag. I have a slight preference for maintaining backward compatibility, but would welcome other opinions.

          Show
          cnauroth Chris Nauroth added a comment - I'm aware of closed-source implementations that may be using the HCFS tests. Really this comes down to compatibility policy for the HCFS tests. I don't think we've discussed that in the past, and there are currently no audience/stability annotations. If we are committed to backward compatibility, then we're better off with the approach I took in patch 001. If we are not committed to backward compatibility for the HCFS tests, then we're free to change anything we want and repurpose is-blobstore as a purely descriptive flag. I have a slight preference for maintaining backward compatibility, but would welcome other opinions.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          What external object stores are we thinking of? We know GCS connector doesn't have those tests, ADL and WASB are in the code, Allyun is coming in. leaving only: a fork of swift (which probably doesn't run these tests) and EMR S3.

          Show
          stevel@apache.org Steve Loughran added a comment - What external object stores are we thinking of? We know GCS connector doesn't have those tests, ADL and WASB are in the code, Allyun is coming in. leaving only: a fork of swift (which probably doesn't run these tests) and EMR S3.
          Hide
          cnauroth Chris Nauroth added a comment -

          Can we keep is-blobstore purely as a high level flag in those *.xml files?

          It sounds like you'd like to see is-blobstore kept as a purely descriptive flag. I can see the appeal of that, but unfortunately, I think it would conflict with what I tried to achieve for backward compatibility in the 001 patch.

          Some of the tests were changed to check the new flags, but also continue checking the is-blobstore flag as a fallback. This helps for backward compatibility if there is a file system implementation outside of the Hadoop source tree that has subclassed AbstractContractCreateTest to run those contract tests in its own project. If we keep is-blobstore in the XML files, then I also need to remove the fallbacks check from the code, because I want WASB and S3A to run the tests with the stricter checks enforced.

          Overall, I'd prefer to stick with the approach in the 001 patch, but let me know your thoughts.

          Show
          cnauroth Chris Nauroth added a comment - Can we keep is-blobstore purely as a high level flag in those *.xml files? It sounds like you'd like to see is-blobstore kept as a purely descriptive flag. I can see the appeal of that, but unfortunately, I think it would conflict with what I tried to achieve for backward compatibility in the 001 patch. Some of the tests were changed to check the new flags, but also continue checking the is-blobstore flag as a fallback. This helps for backward compatibility if there is a file system implementation outside of the Hadoop source tree that has subclassed AbstractContractCreateTest to run those contract tests in its own project. If we keep is-blobstore in the XML files, then I also need to remove the fallbacks check from the code, because I want WASB and S3A to run the tests with the stricter checks enforced. Overall, I'd prefer to stick with the approach in the 001 patch, but let me know your thoughts.
          Hide
          xyao Xiaoyu Yao added a comment -

          Chris Nauroth, the changes looks pretty good to me. I have one question about:

          Deprecated the is-blobstore flag, but retained it in case file system implementations outside the Hadoop source tree are using it. (Side note: do we need to add audience and stability annotations to the contract test classes?)

          Can we keep is-blobstore purely as a high level flag in those *.xml files? The name itself matches with the backstore of those file system.
          We can still use the new flags to differentiate hadoop contract test.

          Show
          xyao Xiaoyu Yao added a comment - Chris Nauroth , the changes looks pretty good to me. I have one question about: Deprecated the is-blobstore flag, but retained it in case file system implementations outside the Hadoop source tree are using it. (Side note: do we need to add audience and stability annotations to the contract test classes?) Can we keep is-blobstore purely as a high level flag in those *.xml files? The name itself matches with the backstore of those file system. We can still use the new flags to differentiate hadoop contract test.
          Hide
          cnauroth Chris Nauroth added a comment -

          All whitespace warnings are in a file I didn't touch with this patch, so they aren't relevant.

          Show
          cnauroth Chris Nauroth added a comment - All whitespace warnings are in a file I didn't touch with this patch, so they aren't relevant.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 1m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 44s branch-2 passed
          +1 compile 5m 50s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 33s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 28s branch-2 passed
          +1 mvnsite 2m 5s branch-2 passed
          +1 mvneclipse 1m 54s branch-2 passed
          +1 findbugs 3m 23s branch-2 passed
          +1 javadoc 1m 30s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 49s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 38s the patch passed
          +1 compile 5m 33s the patch passed with JDK v1.8.0_101
          +1 javac 5m 33s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_111
          +1 javac 6m 32s the patch passed
          +1 checkstyle 1m 28s the patch passed
          +1 mvnsite 2m 16s the patch passed
          +1 mvneclipse 1m 9s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 20s the patch passed
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 0s the patch passed with JDK v1.7.0_111
          +1 unit 7m 45s hadoop-common in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 20s hadoop-openstack in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 28s hadoop-aws in the patch passed with JDK v1.7.0_111.
          +1 unit 1m 46s hadoop-azure in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          105m 40s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13502
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831373/HADOOP-13502-branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b22a87e1699f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 6f37441
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 1m 12s Maven dependency ordering for branch +1 mvninstall 6m 44s branch-2 passed +1 compile 5m 50s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 33s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 28s branch-2 passed +1 mvnsite 2m 5s branch-2 passed +1 mvneclipse 1m 54s branch-2 passed +1 findbugs 3m 23s branch-2 passed +1 javadoc 1m 30s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 49s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 5m 33s the patch passed with JDK v1.8.0_101 +1 javac 5m 33s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_111 +1 javac 6m 32s the patch passed +1 checkstyle 1m 28s the patch passed +1 mvnsite 2m 16s the patch passed +1 mvneclipse 1m 9s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 20s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 0s the patch passed with JDK v1.7.0_111 +1 unit 7m 45s hadoop-common in the patch passed with JDK v1.7.0_111. +1 unit 0m 20s hadoop-openstack in the patch passed with JDK v1.7.0_111. +1 unit 0m 28s hadoop-aws in the patch passed with JDK v1.7.0_111. +1 unit 1m 46s hadoop-azure in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 105m 40s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13502 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831373/HADOOP-13502-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b22a87e1699f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 6f37441 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10645/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch 001. Summary:

          • Introduced 2 new specific contract options: create-overwrites-directory and create-visibility-delayed.
          • Deprecated the is-blobstore flag, but retained it in case file system implementations outside the Hadoop source tree are using it. (Side note: do we need to add audience and stability annotations to the contract test classes?)
          • Cleaned up a few minor JavaDoc omissions in ContractOptions.
          • Updated contract XML configuration files to remove usage of is-blobstore for the actively maintained file systems. Notice that S3A adds create-visibility-delayed, but does not add create-overwrites-directory. That's because HADOOP-13188 recently changed the implementation to avoid overwriting directories.

          So far, I have tested by running all subclasses of AbstractContractCreateTest, except TestSwiftContractCreate. Maybe this will be the patch where I finally get myself set up to run Swift tests.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch 001. Summary: Introduced 2 new specific contract options: create-overwrites-directory and create-visibility-delayed. Deprecated the is-blobstore flag, but retained it in case file system implementations outside the Hadoop source tree are using it. (Side note: do we need to add audience and stability annotations to the contract test classes?) Cleaned up a few minor JavaDoc omissions in ContractOptions . Updated contract XML configuration files to remove usage of is-blobstore for the actively maintained file systems. Notice that S3A adds create-visibility-delayed, but does not add create-overwrites-directory. That's because HADOOP-13188 recently changed the implementation to avoid overwriting directories. So far, I have tested by running all subclasses of AbstractContractCreateTest , except TestSwiftContractCreate . Maybe this will be the patch where I finally get myself set up to run Swift tests.
          Hide
          cnauroth Chris Nauroth added a comment -

          HADOOP-13501 shows that a file system backed by a blob store (WASB) can pass these tests, depending on its implementation.

          Show
          cnauroth Chris Nauroth added a comment - HADOOP-13501 shows that a file system backed by a blob store (WASB) can pass these tests, depending on its implementation.

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development