Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      The DeleteObjectsRequest was not part of the initial S3 API, but was added later. This patch allows one to configure s3a to replace each multidelete request by consecutive single deletes. Evidently, this setting is disabled by default as this causes slower deletes.

      The main motivation is to enable legacy S3-compatible object stores to make the transition from s3n (which does not use multidelete) to s3a, fully allowing the planned s3n deprecation.

      1. HADOOP-12292-001.patch
        7 kB
        Thomas Demoor
      2. HADOOP-12292-002.patch
        7 kB
        Thomas Demoor
      3. HADOOP-12292-003.patch
        7 kB
        Thomas Demoor
      4. HADOOP-12292-004.patch
        12 kB
        Steve Loughran
      5. HADOOP-12292-005.patch
        12 kB
        Steve Loughran
      6. HADOOP-12292-branch-2-005.patch
        12 kB
        Steve Loughran

        Activity

        Hide
        stevel@apache.org Steve Loughran added a comment -

        There's actually one more way to do deletes: asynchronously (thanks to Nick Dimiduk for pointing this out to me). AWS S3 lets you specify an expiry time for data: if you PUT a new lifecycle specification into the bucket which declares everything under a path as expired, then AWS does the cleanup for you, async.

        This argues for having an asyncDelete operation on object stores, which implementations are free to implement themselves (and which could be changed on an endpoint by endpoint basis)

        Show
        stevel@apache.org Steve Loughran added a comment - There's actually one more way to do deletes: asynchronously (thanks to Nick Dimiduk for pointing this out to me). AWS S3 lets you specify an expiry time for data: if you PUT a new lifecycle specification into the bucket which declares everything under a path as expired, then AWS does the cleanup for you, async. This argues for having an asyncDelete operation on object stores, which implementations are free to implement themselves (and which could be changed on an endpoint by endpoint basis)
        Hide
        Thomas Demoor Thomas Demoor added a comment -

        Oh, now I get what you (Steve Loughran and Nick Dimiduk) meant by TTL: Object expiration through bucket lifecylces.

        Not sure that approach is easy, there are several non-trivial issues. Some that immediately come to mind:

        • You are limited to 1000 policy rules per bucket
        • Prefix based
          PUT Object: mybucket/object -> write a file
          PUT Bucket lifecycle: mybucket, Expiration, 1 day, prefix=object -> asynchronously delete this file
          PUT Object: mybucket/object2 -> write another file
          The next day BOTH files are automatically deleted (prefix!!!)
          Also, all future writes which share the prefix will also be deleted automatically after a day.
        Show
        Thomas Demoor Thomas Demoor added a comment - Oh, now I get what you ( Steve Loughran and Nick Dimiduk ) meant by TTL: Object expiration through bucket lifecylces. Not sure that approach is easy, there are several non-trivial issues. Some that immediately come to mind: You are limited to 1000 policy rules per bucket Prefix based PUT Object: mybucket/object -> write a file PUT Bucket lifecycle: mybucket, Expiration, 1 day, prefix=object -> asynchronously delete this file PUT Object: mybucket/object2 -> write another file The next day BOTH files are automatically deleted (prefix!!!) Also, all future writes which share the prefix will also be deleted automatically after a day.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 22s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 39s There were no new javac warning messages.
        +1 javadoc 10m 26s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 1s Site still builds.
        -1 checkstyle 1m 28s The applied patch generated 2 new checkstyle issues (total was 62, now 63).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 11s Tests passed in hadoop-common.
        +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.
            70m 52s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748402/HADOOP-12292-001.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / 90b5104
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/diffcheckstylehadoop-aws.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/testrun_hadoop-aws.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 1s Site still builds. -1 checkstyle 1m 28s The applied patch generated 2 new checkstyle issues (total was 62, now 63). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 11s Tests passed in hadoop-common. +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.     70m 52s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748402/HADOOP-12292-001.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / 90b5104 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/diffcheckstylehadoop-aws.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/testrun_hadoop-common.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7398/console This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        core patch looks good. I can't help thinking, though, it may be cleaner to have a ".enable" property, set to true by default. having a .disable property makes it look like some retrospective fix (which it may be, but we don't need to be blatant about it)

        Show
        stevel@apache.org Steve Loughran added a comment - core patch looks good. I can't help thinking, though, it may be cleaner to have a ".enable" property, set to true by default. having a .disable property makes it look like some retrospective fix (which it may be, but we don't need to be blatant about it)
        Hide
        Thomas Demoor Thomas Demoor added a comment -

        I agree we'd rather not be blatant So, 002.patch implements your suggestion Steve Loughran.

        Show
        Thomas Demoor Thomas Demoor added a comment - I agree we'd rather not be blatant So, 002.patch implements your suggestion Steve Loughran .
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 21m 6s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 51s There were no new javac warning messages.
        +1 javadoc 10m 2s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 6s Site still builds.
        -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 61, now 62).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 23m 19s Tests passed in hadoop-common.
        +1 tools/hadoop tests 0m 15s Tests passed in hadoop-aws.
            72m 14s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761916/HADOOP-12292-002.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / c890c51
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/diffcheckstylehadoop-aws.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/testrun_hadoop-aws.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 6s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 site 3m 6s Site still builds. -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 61, now 62). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 19s Tests passed in hadoop-common. +1 tools/hadoop tests 0m 15s Tests passed in hadoop-aws.     72m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761916/HADOOP-12292-002.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / c890c51 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/diffcheckstylehadoop-aws.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/testrun_hadoop-common.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7691/console This message was automatically generated.
        Hide
        Thomas Demoor Thomas Demoor added a comment -

        Note to self: rerun test-patch, even if you only changed comments.

        The checkstyle issue is deliberate, it follows the style used in this class.

        Show
        Thomas Demoor Thomas Demoor added a comment - Note to self: rerun test-patch, even if you only changed comments. The checkstyle issue is deliberate, it follows the style used in this class.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 57s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 58s There were no new javac warning messages.
        +1 javadoc 10m 4s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 6s Site still builds.
        -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 61, now 62).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 51s Tests passed in hadoop-common.
        +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.
            71m 42s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762136/HADOOP-12292-003.patch
        Optional Tests javadoc javac unit findbugs checkstyle site
        git revision trunk / ead1b9e
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/diffcheckstylehadoop-aws.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/testrun_hadoop-aws.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 6s Site still builds. -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 61, now 62). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 51s Tests passed in hadoop-common. +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.     71m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762136/HADOOP-12292-003.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / ead1b9e checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/diffcheckstylehadoop-aws.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/testrun_hadoop-common.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7700/console This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Looking at this. It doesn't come with a test, does it? I'll do one by subclassing an existing test and flipping the bit

        Show
        stevel@apache.org Steve Loughran added a comment - Looking at this. It doesn't come with a test, does it? I'll do one by subclassing an existing test and flipping the bit
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 25s Maven dependency ordering for branch
        +1 mvninstall 6m 31s trunk passed
        +1 compile 5m 52s trunk passed with JDK v1.8.0_66
        +1 compile 6m 49s trunk passed with JDK v1.7.0_91
        +1 checkstyle 1m 5s trunk passed
        +1 mvnsite 1m 23s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 2m 3s trunk passed
        +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 18s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 0m 53s the patch passed
        +1 compile 5m 58s the patch passed with JDK v1.8.0_66
        +1 javac 5m 58s the patch passed
        +1 compile 6m 49s the patch passed with JDK v1.7.0_91
        +1 javac 6m 49s the patch passed
        -1 checkstyle 1m 4s root: patch generated 2 new + 48 unchanged - 1 fixed = 50 total (was 49)
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 2m 26s the patch passed
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 18s the patch passed with JDK v1.7.0_91
        -1 unit 16m 24s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_66.
        -1 unit 6m 17s hadoop-common in the patch failed with JDK v1.7.0_91.
        +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        73m 36s



        Reason Tests
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
        JDK v1.7.0_91 Failed junit tests hadoop.net.TestClusterTopology



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12762136/HADOOP-12292-003.patch
        JIRA Issue HADOOP-12292
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 1911013d058a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4e5e1c0
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
        Max memory used 77MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 6m 31s trunk passed +1 compile 5m 52s trunk passed with JDK v1.8.0_66 +1 compile 6m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 18s trunk passed with JDK v1.7.0_91 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 5m 58s the patch passed with JDK v1.8.0_66 +1 javac 5m 58s the patch passed +1 compile 6m 49s the patch passed with JDK v1.7.0_91 +1 javac 6m 49s the patch passed -1 checkstyle 1m 4s root: patch generated 2 new + 48 unchanged - 1 fixed = 50 total (was 49) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 26s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 18s the patch passed with JDK v1.7.0_91 -1 unit 16m 24s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_66. -1 unit 6m 17s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 73m 36s Reason Tests JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_91 Failed junit tests hadoop.net.TestClusterTopology Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12762136/HADOOP-12292-003.patch JIRA Issue HADOOP-12292 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 1911013d058a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e5e1c0 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8549/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        This patch makes no change to the src/main of patch -003, just subclasses the TestS3ADeleteManyFiles test to add a test TestS3ADeleteFilesOneByOne for singular operation.

        In the process of running these, the s3a test suite alone proved enough to trigger surefire timeouts for me. I've had to make the surefire timeout in the hadoop build something configurable; surprised it hadn't been done already.

        I've done all the S3 tests; a transient failure of TestS3AFileSystemContract.testLSRootDir which I couldn't repeat was the only issue.

        Let's see what jenkins says, but otherwise the patch LGTM

        Show
        stevel@apache.org Steve Loughran added a comment - This patch makes no change to the src/main of patch -003, just subclasses the TestS3ADeleteManyFiles test to add a test TestS3ADeleteFilesOneByOne for singular operation. In the process of running these, the s3a test suite alone proved enough to trigger surefire timeouts for me. I've had to make the surefire timeout in the hadoop build something configurable; surprised it hadn't been done already. I've done all the S3 tests; a transient failure of TestS3AFileSystemContract.testLSRootDir which I couldn't repeat was the only issue. Let's see what jenkins says, but otherwise the patch LGTM
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Patch -005 with all the changes to the tests checked in.

        Show
        stevel@apache.org Steve Loughran added a comment - Patch -005 with all the changes to the tests checked in.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 22s Maven dependency ordering for branch
        +1 mvninstall 6m 35s trunk passed
        +1 compile 6m 23s trunk passed with JDK v1.8.0_72
        +1 compile 6m 51s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 23s trunk passed
        +1 mvnsite 1m 36s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 13s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 23s Maven dependency ordering for patch
        +1 mvninstall 1m 0s the patch passed
        +1 compile 5m 45s the patch passed with JDK v1.8.0_72
        +1 javac 5m 45s the patch passed
        +1 compile 6m 42s the patch passed with JDK v1.7.0_95
        +1 javac 6m 42s the patch passed
        -1 checkstyle 1m 4s root: patch generated 2 new + 48 unchanged - 1 fixed = 50 total (was 49)
        +1 mvnsite 1m 31s the patch passed
        +1 mvneclipse 0m 35s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project
        +1 findbugs 2m 23s the patch passed
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 28s the patch passed with JDK v1.7.0_95
        +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_72.
        +1 unit 6m 52s hadoop-common in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_95.
        +1 unit 7m 6s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        67m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786569/HADOOP-12292-005.patch
        JIRA Issue HADOOP-12292
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 3262888c62f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c649794
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/artifact/patchprocess/diff-checkstyle-root.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: .
        Max memory used 77MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 6m 23s trunk passed with JDK v1.8.0_72 +1 compile 6m 51s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 23s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 39s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project +1 findbugs 2m 0s trunk passed +1 javadoc 1m 13s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 5m 45s the patch passed with JDK v1.8.0_72 +1 javac 5m 45s the patch passed +1 compile 6m 42s the patch passed with JDK v1.7.0_95 +1 javac 6m 42s the patch passed -1 checkstyle 1m 4s root: patch generated 2 new + 48 unchanged - 1 fixed = 50 total (was 49) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project +1 findbugs 2m 23s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_95 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_72. +1 unit 6m 52s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_95. +1 unit 7m 6s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 67m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786569/HADOOP-12292-005.patch JIRA Issue HADOOP-12292 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3262888c62f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c649794 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: . Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8555/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1, committed to trunk

        I'm going to quickly see if I can backport to branch-2/2.8, which means the patch need to be tuned for the fact that HADOOP-11684 isn't in there —the diff won't apply directly

        Show
        stevel@apache.org Steve Loughran added a comment - +1, committed to trunk I'm going to quickly see if I can backport to branch-2/2.8, which means the patch need to be tuned for the fact that HADOOP-11684 isn't in there —the diff won't apply directly
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9254 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9254/)
        HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via (stevel: rev 29ae25801380b94442253c4202dee782dc4713f5)

        • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3ADeleteManyFiles.java
        • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java
        • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
        • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
        • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-project/pom.xml
        • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9254 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9254/ ) HADOOP-12292 . Make use of DeleteObjects optional. (Thomas Demoor via (stevel: rev 29ae25801380b94442253c4202dee782dc4713f5) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3ADeleteManyFiles.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
        Hide
        stevel@apache.org Steve Loughran added a comment -

        This is the branch-2/2.8 version of the patch, as I'm about to apply to branch-2.8+

        Show
        stevel@apache.org Steve Loughran added a comment - This is the branch-2/2.8 version of the patch, as I'm about to apply to branch-2.8+
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9255 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9255/)
        HADOOP-12292. Make use of DeleteObjects optional -move CHANGES.TXT (stevel: rev 1495ff979a47d17838218da77df4406f94b8227c)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9255 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9255/ ) HADOOP-12292 . Make use of DeleteObjects optional -move CHANGES.TXT (stevel: rev 1495ff979a47d17838218da77df4406f94b8227c) hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 3m 0s Maven dependency ordering for branch
        +1 mvninstall 7m 56s branch-2 passed
        +1 compile 6m 30s branch-2 passed with JDK v1.8.0_66
        +1 compile 6m 56s branch-2 passed with JDK v1.7.0_91
        +1 checkstyle 1m 10s branch-2 passed
        +1 mvnsite 1m 27s branch-2 passed
        +1 mvneclipse 1m 16s branch-2 passed
        0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project
        -1 findbugs 1m 44s hadoop-common-project/hadoop-common in branch-2 has 5 extant Findbugs warnings.
        +1 javadoc 1m 24s branch-2 passed with JDK v1.8.0_66
        +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_91
        0 mvndep 0m 24s Maven dependency ordering for patch
        +1 mvninstall 1m 9s the patch passed
        +1 compile 6m 17s the patch passed with JDK v1.8.0_66
        +1 javac 6m 17s the patch passed
        +1 compile 6m 51s the patch passed with JDK v1.7.0_91
        +1 javac 6m 51s the patch passed
        -1 checkstyle 1m 7s root: patch generated 2 new + 56 unchanged - 1 fixed = 58 total (was 57)
        +1 mvnsite 1m 21s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        -1 whitespace 0m 0s The patch has 61 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project
        +1 findbugs 2m 38s the patch passed
        +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 35s the patch passed with JDK v1.7.0_91
        +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_66.
        -1 unit 16m 59s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.7.0_91.
        +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        85m 1s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.fs.TestSymlinkLocalFSFileSystem
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786681/HADOOP-12292-branch-2-005.patch
        JIRA Issue HADOOP-12292
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 82e0066caa30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / e4c01b8
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: .
        Max memory used 74MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 3m 0s Maven dependency ordering for branch +1 mvninstall 7m 56s branch-2 passed +1 compile 6m 30s branch-2 passed with JDK v1.8.0_66 +1 compile 6m 56s branch-2 passed with JDK v1.7.0_91 +1 checkstyle 1m 10s branch-2 passed +1 mvnsite 1m 27s branch-2 passed +1 mvneclipse 1m 16s branch-2 passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-project -1 findbugs 1m 44s hadoop-common-project/hadoop-common in branch-2 has 5 extant Findbugs warnings. +1 javadoc 1m 24s branch-2 passed with JDK v1.8.0_66 +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_91 0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 6m 17s the patch passed with JDK v1.8.0_66 +1 javac 6m 17s the patch passed +1 compile 6m 51s the patch passed with JDK v1.7.0_91 +1 javac 6m 51s the patch passed -1 checkstyle 1m 7s root: patch generated 2 new + 56 unchanged - 1 fixed = 58 total (was 57) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 37s the patch passed -1 whitespace 0m 0s The patch has 61 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-project +1 findbugs 2m 38s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 35s the patch passed with JDK v1.7.0_91 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_66. -1 unit 16m 59s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66. +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.7.0_91. +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 85m 1s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.TestSymlinkLocalFSFileSystem JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786681/HADOOP-12292-branch-2-005.patch JIRA Issue HADOOP-12292 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 82e0066caa30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e4c01b8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: . Max memory used 74MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8562/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Thomas Demoor Thomas Demoor added a comment -

        Thanks Steve Loughran for adding the test and committing. Saw it this afternoon and launched a test run. By the time I got home, it was committed. For the record, the test came out fine vs EU-west1 on my end:

        {{ Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteFilesOneByOne
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 666.261 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteFilesOneByOne }}

        I think it would make our life easier to backport HADOOP-11684 in branch-2. It fixes an obvious bug (see f.i. HADOOP-12319). Can we add a deprecation warning or take other actions to make it branch-2 material?

        Show
        Thomas Demoor Thomas Demoor added a comment - Thanks Steve Loughran for adding the test and committing. Saw it this afternoon and launched a test run. By the time I got home, it was committed. For the record, the test came out fine vs EU-west1 on my end: {{ Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteFilesOneByOne Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 666.261 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteFilesOneByOne }} I think it would make our life easier to backport HADOOP-11684 in branch-2. It fixes an obvious bug (see f.i. HADOOP-12319 ). Can we add a deprecation warning or take other actions to make it branch-2 material?
        Hide
        stevel@apache.org Steve Loughran added a comment -

        I saw that, was going to cherry pick in the HADOOP-11684 branch, though I did see the discussion. I'd propose raising a JIRA on backporting, push it into 2.9 and document as a change. S3a is still stabilising, after all

        Show
        stevel@apache.org Steve Loughran added a comment - I saw that, was going to cherry pick in the HADOOP-11684 branch, though I did see the discussion. I'd propose raising a JIRA on backporting, push it into 2.9 and document as a change. S3a is still stabilising, after all

          People

          • Assignee:
            Thomas Demoor Thomas Demoor
            Reporter:
            Thomas Demoor Thomas Demoor
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development