Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.1.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      S3AFileSystem to add support for a special path, such as .temp_pending_put/ or similar, which, when used as the base of a path, indicates that the file is actually to be saved to the parent dir, but only via a delayed put commit operation.

      At the same time, we may need blocks on some normal fileIO ops under these dirs, especially rename and delete, as this would cause serious problems including data loss and large bills for pending data.

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          +when creating a path .temp_pending_put/dest.txt, there must be no existing entry there. we will need to rely on a consistent FS for this, and may also want to do some PUT operations which bypass the s3a FS APIs, but still work with s3guard or an existing consistent object store. (e.g an atomic PUT/mkdir.

          Show
          stevel@apache.org Steve Loughran added a comment - +when creating a path .temp_pending_put/dest.txt , there must be no existing entry there. we will need to rely on a consistent FS for this, and may also want to do some PUT operations which bypass the s3a FS APIs, but still work with s3guard or an existing consistent object store. (e.g an atomic PUT/mkdir.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development