Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-17498

Add a MkdirOperation for chained S3 operations during mkdir

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.1
    • Fix Version/s: None
    • Component/s: fs/s3

      Description

      on S3A mkdirs is implemented as a walk up the tree looking for any parent dir, exiting fast when found; failing if a file is found.

      Proposed: pull this out into its own MkDir operation with the goals of efficiency, track duration and isolate/test

      • specific callbacks to getFileStatus and createFakeDirectory(key);
      • Move from simple getFileStatus call to probes assuming dir exists before looking for a file: HEAD + path /, LIST path,
      • the probe for a file HEAD path will only succeed in a failure mode where higher cost is a detail

      I think I'd also like to start looking at what we can do with some context tracking/telemetry where something provides an operation ID and ability to attach this to the requests

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m