Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-15183

S3Guard store becomes inconsistent after partial failure of rename

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.0.0
    • 3.3.0
    • fs/s3
    • None

    Description

      If an S3A rename() operation fails partway through, such as when the user doesn't have permissions to delete the source files after copying to the destination, then the s3guard view of the world ends up inconsistent. In particular the sequence

      (assuming src/file* is a list of files file1...file10 and read only to caller)

      1. create file rename src/file1 dest/ ; expect AccessDeniedException in the delete, dest/file1 will exist
      2. delete file dest/file1
      3. rename src/file* dest/ ; expect failure
      4. list dest; you will not see dest/file1

      You will not see file1 in the listing, presumably because it will have a tombstone marker and the update at the end of the rename() didn't take place: the old data is still there.

      Attachments

        1. HADOOP-15183-001.patch
          23 kB
          Steve Loughran
        2. HADOOP-15183-002.patch
          33 kB
          Steve Loughran
        3. org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
          203 kB
          Steve Loughran
        4. rmtestfailure.txt
          50 kB
          Steve Loughran

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16390 Build fails due to bad use of '>' in javadoc

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-15604 Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15658 Memory leak in S3AOutputStream

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16364 S3Guard table destroy to map IllegalArgumentExceptions to IOEs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          depends upon

          Sub-task - The sub-task of the issue HADOOP-15176 Enhance IAM Assumed Role support in S3A client

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-13936 S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete()

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-14936 S3Guard: remove "experimental" from documentation

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15492 increase performance of s3guard import command

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-16380 S3A tombstones can confuse empty directory status

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16430 S3AFilesystem.delete to incrementally update s3guard with deletions

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17869 fs.s3a.connection.maximum should be bigger than fs.s3a.threads.max

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #654

          Web Link GitHub Pull Request #835

          Web Link GitHub Pull Request #843

          Web Link GitHub Pull Request #951

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: