[FLINK-23127] Can't use plugins for GCS filesystem

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.13.0
• Component/s: FileSystems
• Assignee: Unassigned
• Reporter: Yaroslav Tkachenko
• Created: 23/Jun/21 16:48
• Updated: 24/Jun/21 07:01

Description

I've been trying to add support for the GCS filesystem. I have a working example where I add two JARs to the /opt/flink/lib/ folder:

• GCS Hadoop connector
• Shaded Hadoop using flink-shaded-hadoop-2-uber-2.8.3-10.0.jar

Now I'm trying to follow the advice from this page and use Plugins instead. I followed the recommendations from here. Now I have two JARs in the /opt/flink/plugins/hadoop-gcs/ folder:

• GCS Hadoop connector
• Non-shaded Hadoop using hadoop-common-2.10.1.jar

As I can see, shading is not required for plugins (that's one of the reasons to use them), so I want to make it work with a simple non-shaded hadoop-common.

However, the JobManager fails with an exception (full stacktrace is available is an attachment):

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

The exception is thrown when org.apache.hadoop.conf.Configuration and org.apache.hadoop.fs.FileSystem are not available in the classpath, but they're available in hadoop-common and should have been loaded.