Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13786

Add S3A committers for zero-rename commits to S3 endpoints

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.1.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).

      I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.

        Attachments

        1. cloud-intergration-test-failure.log
          334 kB
          Ewan Higgs
        2. HADOOP-13786-036.patch
          778 kB
          Steve Loughran
        3. HADOOP-13786-037.patch
          851 kB
          Steve Loughran
        4. HADOOP-13786-038.patch
          854 kB
          Steve Loughran
        5. HADOOP-13786-039.patch
          929 kB
          Steve Loughran
        6. HADOOP-13786-HADOOP-13345-001.patch
          126 kB
          Steve Loughran
        7. HADOOP-13786-HADOOP-13345-002.patch
          150 kB
          Steve Loughran
        8. HADOOP-13786-HADOOP-13345-003.patch
          170 kB
          Steve Loughran
        9. HADOOP-13786-HADOOP-13345-004.patch
          222 kB
          Steve Loughran
        10. HADOOP-13786-HADOOP-13345-005.patch
          225 kB
          Steve Loughran
        11. HADOOP-13786-HADOOP-13345-006.patch
          231 kB
          Steve Loughran
        12. HADOOP-13786-HADOOP-13345-006.patch
          231 kB
          Steve Loughran
        13. HADOOP-13786-HADOOP-13345-007.patch
          246 kB
          Steve Loughran
        14. HADOOP-13786-HADOOP-13345-009.patch
          454 kB
          Steve Loughran
        15. HADOOP-13786-HADOOP-13345-010.patch
          471 kB
          Steve Loughran
        16. HADOOP-13786-HADOOP-13345-011.patch
          471 kB
          Steve Loughran
        17. HADOOP-13786-HADOOP-13345-012.patch
          484 kB
          Steve Loughran
        18. HADOOP-13786-HADOOP-13345-013.patch
          487 kB
          Steve Loughran
        19. HADOOP-13786-HADOOP-13345-015.patch
          499 kB
          Steve Loughran
        20. HADOOP-13786-HADOOP-13345-016.patch
          501 kB
          Steve Loughran
        21. HADOOP-13786-HADOOP-13345-017.patch
          505 kB
          Steve Loughran
        22. HADOOP-13786-HADOOP-13345-018.patch
          512 kB
          Steve Loughran
        23. HADOOP-13786-HADOOP-13345-019.patch
          521 kB
          Steve Loughran
        24. HADOOP-13786-HADOOP-13345-020.patch
          549 kB
          Steve Loughran
        25. HADOOP-13786-HADOOP-13345-021.patch
          558 kB
          Steve Loughran
        26. HADOOP-13786-HADOOP-13345-022.patch
          564 kB
          Steve Loughran
        27. HADOOP-13786-HADOOP-13345-023.patch
          597 kB
          Steve Loughran
        28. HADOOP-13786-HADOOP-13345-024.patch
          604 kB
          Steve Loughran
        29. HADOOP-13786-HADOOP-13345-025.patch
          624 kB
          Steve Loughran
        30. HADOOP-13786-HADOOP-13345-026.patch
          656 kB
          Steve Loughran
        31. HADOOP-13786-HADOOP-13345-027.patch
          653 kB
          Steve Loughran
        32. HADOOP-13786-HADOOP-13345-028.patch
          669 kB
          Steve Loughran
        33. HADOOP-13786-HADOOP-13345-028.patch
          669 kB
          Steve Loughran
        34. HADOOP-13786-HADOOP-13345-029.patch
          674 kB
          Steve Loughran
        35. HADOOP-13786-HADOOP-13345-030.patch
          690 kB
          Steve Loughran
        36. HADOOP-13786-HADOOP-13345-031.patch
          704 kB
          Steve Loughran
        37. HADOOP-13786-HADOOP-13345-032.patch
          733 kB
          Steve Loughran
        38. HADOOP-13786-HADOOP-13345-033.patch
          748 kB
          Steve Loughran
        39. HADOOP-13786-HADOOP-13345-035.patch
          771 kB
          Steve Loughran
        40. MAPREDUCE-6823-003.patch
          44 kB
          Steve Loughran
        41. objectstore.pdf
          187 kB
          Steve Loughran
        42. s3committer-master.zip
          42 kB
          Ryan Blue

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                11 Vote for this issue
                Watchers:
                44 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: