Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-15183

S3Guard store becomes inconsistent after partial failure of rename

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.3.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      If an S3A rename() operation fails partway through, such as when the user doesn't have permissions to delete the source files after copying to the destination, then the s3guard view of the world ends up inconsistent. In particular the sequence

      (assuming src/file* is a list of files file1...file10 and read only to caller)

      1. create file rename src/file1 dest/ ; expect AccessDeniedException in the delete, dest/file1 will exist
      2. delete file dest/file1
      3. rename src/file* dest/ ; expect failure
      4. list dest; you will not see dest/file1

      You will not see file1 in the listing, presumably because it will have a tombstone marker and the update at the end of the rename() didn't take place: the old data is still there.

        Attachments

        1. HADOOP-15183-001.patch
          23 kB
          Steve Loughran
        2. HADOOP-15183-002.patch
          33 kB
          Steve Loughran
        3. org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
          203 kB
          Steve Loughran
        4. rmtestfailure.txt
          50 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: