[HADOOP-16150] checksumFS doesn't wrap concat(): concatenated files don't have checksums

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: fs
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/Feb/19 19:07
• Updated: 05/Mar/19 22:24
• Resolved: 05/Mar/19 18:37

Description

Followon from HADOOP-16107. FilterFS passes through the concat operation, and checksum FS doesn't override that call -so files created through concat do not have checksums.

If people are using a checksummed fs directly with the expectations that they will, that expectation is not being met.

What to do?

• fail always?
• fail if checksums are enabled?
• try and implement the concat operation from raw local up at the checksum level

append() just gives up always; doing the same for concat would be the simplest. Again, brings us back to "need a way to see if an FS supports a feature before invocation", here checksum fs would reject append and concat

Issue Links

• is caused by: HDFS-13186 [PROVIDED Phase 2] Multipart Uploader API
• is related to: HDFS-13934 Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader
• relates to: HADOOP-16107 FilterFileSystem doesn't wrap all create() or new builder calls; may skip CRC logic
• relates to: HADOOP-15691 Add PathCapabilities to FS and FC to complement StreamCapabilities
• links to: GitHub Pull Request #525