Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      There's a TODO in dirListingUnion:

      // TODO optimize for when allowAuthoritative = false

      There will be cases when we can intelligently avoid a round trip: if S3A results are a subset or the metadatastore results (including them being equal or empty) then writing back will do nothing (although perhaps that should set the authoritative flag if it isn't set already).

      There may also be cases where users want to just skip that altogether. It's wasted work if authoritative mode is disabled, so perhaps we want to trigger a skip if that's false, or perhaps it should be a separate property. First one makes for simpler config, second is more flexible...

        Attachments

        1. HADOOP-14020-HADOOP-13345.004.patch
          8 kB
          Sean Mackrory
        2. HADOOP-14020-HADOOP-13345.003.patch
          8 kB
          Sean Mackrory
        3. HADOOP-14020-HADOOP-13345.002.patch
          8 kB
          Sean Mackrory
        4. HADOOP-14020-HADOOP-13345.001.patch
          11 kB
          Sean Mackrory

          Issue Links

            Activity

              People

              • Assignee:
                mackrorysd Sean Mackrory
                Reporter:
                mackrorysd Sean Mackrory
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: