Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-13843

S3Guard, MetadataStore to support atomic create(path, overwrite=false)

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Support atomically enforced file creation. Current s3a can do a check in create() and fail if there is something there, but a new entry only gets created at the end of the PUT; during the entire interval between that check and the close() of the stream, there's nothing to stop other callers creating an object.

      Proposed: s3afs can do a check + create a 0 byte file at the path; that'd need some putNoOverwrite(DirListingMetadata) call in MetadataStore, followed by a PUT of an 0-byte file to S3. That will increase cost of file creation, though at least with the MD store, the cost of the initial getFileStatus() check is down.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: