[HADOOP-14402] roll out StreamCapabilities across output streams of all filesystems

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 3.3.1
• Component/s: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 09/May/17 15:43
• Updated: 10/Feb/21 14:59
• Resolved: 10/Feb/21 14:58

Description

HDFS-11644 added a way to probe streams for capabilities, with dfsoutput stream implementing some.

We should roll this out to all our output streams, with capabilities to define

• is metadata in sync with flushed data (HDFS: no)
• does LocalFileSystem really do a sync/flush (as I don't think any subclass of ChecksumFS does, so FSOutputSummer should declare this in its capabilities
• are intermediate writes visible at all?
• Is close() potentially a long operation (for object stores, yes, on a case-by-case basis)

We'll need useful names for these options, obviously, tests, etc etc

Issue Links

• depends upon: HDFS-11644 Support for querying outputstream capabilities
• Is contained by: HADOOP-13327 Add OutputStream + Syncable to the Filesystem Specification
• is duplicated by: HDFS-12263 Revise StreamCapacities doc to describe the API usage and the requirements for customized OutputStream implemetation
• is related to: HDFS-12251 Add document for StreamCapabilities
• is related to: HDFS-12263 Revise StreamCapacities doc to describe the API usage and the requirements for customized OutputStream implemetation
• relates to: HADOOP-9565 Add a Blobstore interface to add to blobstore FileSystems

Sub-Tasks

• Add StreamCapabilities support to s3a: Resolved
• Add StreamCapabilities support to ADLS: Resolved
• Add StreamCapabilities support to WASB: Resolved
• Add StreamCapabilities support to Swift: Open
• Add StreamCapabilities support to Aliyun OSS: Open