Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11520

Clean incomplete multi-part uploads in S3A tests

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload.

      People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart).

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695622/HADOOP-11520.001.patch
          against trunk revision ffc75d6.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//artifact/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695622/HADOOP-11520.001.patch against trunk revision ffc75d6. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5555//console This message is automatically generated.
          Hide
          thodemoor Thomas Demoor added a comment -

          unrelated error. photoshop documents lol?

          Show
          thodemoor Thomas Demoor added a comment - unrelated error. photoshop documents lol?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1, committed to branch-2. thanks!

          Show
          stevel@apache.org Steve Loughran added a comment - +1, committed to branch-2. thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7044 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7044/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7044 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7044/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/831/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/831/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/)
          HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/ ) HADOOP-11520 . Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/ )
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/)

          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/ )

            People

            • Assignee:
              thodemoor Thomas Demoor
              Reporter:
              thodemoor Thomas Demoor
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development