XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 3.3.1
    • Component/s: fs/s3
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      The S3A connector now has an option to stop deleting directory markers as files are written. This eliminates the IO throttling the operations can cause, and avoids creating tombstone markers on versioned S3 buckets.

      This feature is incompatible with all versions of Hadoop which lack the HADOOP-17199 change to list and getFileStatus calls.

      Consult the S3A documentation for further details
      Show
      The S3A connector now has an option to stop deleting directory markers as files are written. This eliminates the IO throttling the operations can cause, and avoids creating tombstone markers on versioned S3 buckets. This feature is incompatible with all versions of Hadoop which lack the HADOOP-17199 change to list and getFileStatus calls. Consult the S3A documentation for further details

      Description

      Users of s3a may not realize that, in some cases, it does not interoperate well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558).

      Specifically, if a user:

      • Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
      • Copies data into that directory via another tool, i.e. aws cli.
      • Tries to access the data in that directory with any Hadoop software.

      Then the last step fails because the fake empty directory blob that s3a wrote in the first step, causes s3a (listStatus() etc.) to continue to treat that directory as empty, even though the second step was supposed to populate the directory with data.

      I wanted to document this fact for users. We may mark this as not-fix, "by design".. May also be interesting to brainstorm solutions and/or a config option to change the behavior if folks care.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                fabbri Aaron Fabbri
              • Votes:
                0 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m