[HADOOP-9565] Add a Blobstore interface to add to blobstore FileSystems

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: fs, fs/s3, fs/swift
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 15/May/13 23:09
• Updated: 22/Oct/19 14:33

Description

We can make the fact that some FileSystem implementations are really blobstores, with different atomicity and consistency guarantees, by adding a Blobstore interface to add to them.

This could also be a place to add a Copy(Path,Path) method, assuming that all blobstores implement at server-side copy operation as a substitute for rename.

Issue Links

• is depended upon by: HADOOP-11556 SwiftNativeFileSystem rename being interrupted during "fs -put"
• is depended upon by: HADOOP-9577 Actual data loss using s3n (against US Standard region)
• is depended upon by: HADOOP-11487 FileNotFound on distcp to s3n/s3a due to creation inconsistency
• is duplicated by: HADOOP-11525 FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.
• is related to: HADOOP-11867 Add a high-performance vectored read API.
• is related to: HDFS-11704 OzoneFileSystem: A Hadoop file system implementation for Ozone
• is related to: HADOOP-12046 Avoid creating "._COPYING_" temporary file when copying file to Swift file system
• is related to: HADOOP-16604 Provide copy functionality for cloud native applications
• is related to: HADOOP-14402 roll out StreamCapabilities across output streams of all filesystems
• relates to: HDFS-11644 Support for querying outputstream capabilities
• relates to: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
• relates to: HADOOP-11525 FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.