[HADOOP-15691] Add PathCapabilities to FS and FC to complement StreamCapabilities

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.2, 3.2.3
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 23/Aug/18 17:49
• Updated: 10/Jun/22 16:50
• Resolved: 25/Sep/19 11:22

Description

This complements the StreamCapabilities interface by allowing applications
to probe for a specific path on a specific instance of a FileSystem
or FileContext offering a specific feature.

This is intended to allow applications to determine

• Whether a method is implemented before calling it and dealing with
  any subsequent UnsupportedOperationException.
• Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .

The plan is to identify and document more capabilities -and for file systems
which add new features, for a declaration of the availability of the feature to
always be available.

The interface may be offered by other classes too; there is no restriction here.

Note

• The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
• Permissions are not checked.

This is needed for

• HADOOP-14707: declare that a dest FS supports permissions
• object stores to declare that they offer PUT-in-place alongside (slow-rename)
• Anything else where the implementation semantics of an FS is so different caller apps would benefit from probing for the underlying semantics

I know, we want all filesystem to work exactly the same. But it doesn't hold, especially for object stores —and to efficiently use them, callers need to be able to ask for specific features.

Issue Links

• is depended upon by: HADOOP-15777 Add support of getSnapshotDiff for all the FS inherit from FileSystem
• is related to: HADOOP-16150 checksumFS doesn't wrap concat(): concatenated files don't have checksums
• is related to: HDFS-14111 hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
• is related to: HADOOP-18287 Provide a shim library for modern FS APIs
• is related to: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• is related to: HADOOP-17210 backport HADOOP-15691 PathCapabilities API to branch-3.2
• is related to: HDFS-13879 FileSystem: Add allowSnapshot, disallowSnapshot, getSnapshotDiffReport and getSnapshottableDirListing
• is required by: YARN-9607 Auto-configuring rollover-size of IFile format for non-appendable filesystems
• relates to: HADOOP-14707 AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
• links to: GitHub Pull Request #566
• links to: GitHub Pull Request #568