XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.5
    • 3.4.0
    • fs/s3
    • Incompatible change
    • Hide
      The s3a connector no longer deletes directory markers by default, which speeds up write operations, reduces iO throttling and saves money. this can cause problems with older hadoop releases trying to write to the same bucket. (Hadoop 3.3.0; Hadoop 3.2.x before 3.2.2, and all previous releases). Set "fs.s3a.directory.marker.retention" to "delete" for backwards compatibility
      Show
      The s3a connector no longer deletes directory markers by default, which speeds up write operations, reduces iO throttling and saves money. this can cause problems with older hadoop releases trying to write to the same bucket. (Hadoop 3.3.0; Hadoop 3.2.x before 3.2.2, and all previous releases). Set "fs.s3a.directory.marker.retention" to "delete" for backwards compatibility

    Description

      Change the default value of "fs.s3a.directory.marker.retention" to keep; update docs to match.

      maybe include with HADOOP-17802 so we don't blow up with fewer markers being created.

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: