[FLINK-15777] Truncate not working with plugins (as expected)

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 1.9.1, 1.10.0, 1.11.0
• Component/s: FileSystems
• Labels: pull-request-available
• Assignee: Arvid Heise
• Reporter: Arvid Heise
• Created: 27/Jan/20 13:35
• Updated: 22/Jun/21 14:06
• Resolved: 30/Jan/20 08:59

Description

FLINK-14170 introduced Hadoop version check to support older versions with limited functionality. However, the fix checks the Hadoop version of the system classloader and not of the plugin (of the actual used filesystem).

The version checks will fail when there is a Hadoop version in the classpath of Flink, independent of the bundled version of the filesystem.

On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you cannot write to S3 with StreamingFileSink.

Issue Links

• is fixed by: FLINK-16015 Refine fallback filesystems to only handle specific filesystems
• relates to: FLINK-15783 Test release candidate of Flink 1.10 on EMR
• links to: GitHub Pull Request #10952