[FLINK-16015] Refine fallback filesystems to only handle specific filesystems

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.1, 1.11.0
• Fix Version/s: 1.11.0
• Component/s: FileSystems
• Labels: pull-request-available

Release Note

By default, if there is an official filesystem plugin for a given schema, it will not be allowed to use fallback filesystem factories (like HADOOP libraries on the classpath) to load it. Added {{fs.allowed-fallback-filesystems}} configuration option to override this behaviour.

• Assignee: Arvid Heise
• Reporter: Arvid Heise
• Created: 12/Feb/20 10:08
• Updated: 03/Mar/20 09:11
• Resolved: 02/Mar/20 11:49

Description

Currently, if no s3 plugin is included, hadoop is used as a fallback, which introduces a wide variety of problems. We should probably only white list specific protocols that work well (e.g. hdfs).

Issue Links

• fixes: FLINK-15777 Truncate not working with plugins (as expected)
• links to: GitHub Pull Request #11139