[FLINK-9245] Can't create a BucketingSink with a provided Configuration if no hadoop defaults

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.4.2
• Component/s: Connectors / Common
• Assignee: Unassigned
• Reporter: Julien Cuquemelle
• Created: 24/Apr/18 07:47
• Updated: 24/May/18 13:54
• Resolved: 04/May/18 09:27

Description

We build Integration tests using this kind of code: 

val bucketingSink = new BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs)
bucketingSink.setBucketer(new DateTimeBucketer[Row]("yyyy-MM-dd--HHmm"))
outpuStream.addSink(bucketingSink)

Here, the hadoopRule is providing a valid hdfs config that should allows this kind of code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set up, like a developper workstation or a Jenkins slave.

When running this code on such a machine, the .createHadoopFileSystem(...) fails with 

The given file system URI (hdfs:///user/$USER/application_name/) did not describe the authority 
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)

because it tries to instantiate the fileSystem from a default configuration in .getUnguardedFileSystem() ; as the default conf doesn't exist, the default filesystem resolves to "file:///" and the checks of the consistency of the URI fails because no authority can be found So the whole filesystem creation fails before actually trying to use the provided config.

Issue Links

• is duplicated by: FLINK-9244 Can't create a BucketingSink with a provided Configuration if no hadoop defaults