Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      S3AFileSystem to add support for a special path, such as .temp_pending_put/ or similar, which, when used as the base of a path, indicates that the file is actually to be saved to the parent dir, but only via a delayed put commit operation.

      At the same time, we may need blocks on some normal fileIO ops under these dirs, especially rename and delete, as this would cause serious problems including data loss and large bills for pending data.

        Activity

        Hide
        stevel@apache.org Steve Loughran added a comment -

        +when creating a path .temp_pending_put/dest.txt, there must be no existing entry there. we will need to rely on a consistent FS for this, and may also want to do some PUT operations which bypass the s3a FS APIs, but still work with s3guard or an existing consistent object store. (e.g an atomic PUT/mkdir.

        Show
        stevel@apache.org Steve Loughran added a comment - +when creating a path .temp_pending_put/dest.txt , there must be no existing entry there. we will need to rely on a consistent FS for this, and may also want to do some PUT operations which bypass the s3a FS APIs, but still work with s3guard or an existing consistent object store. (e.g an atomic PUT/mkdir.

          People

          • Assignee:
            Unassigned
            Reporter:
            stevel@apache.org Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development