[BEAM-7613] HadoopFileSystem can be only used with fs.defaultFS

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.13.0
• Fix Version/s: 2.16.0
• Component/s: io-java-hadoop-file-system
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 21/Jun/19 10:49
• Updated: 16/May/20 14:11
• Resolved: 08/Aug/19 17:03

Description

HadoopFileSystem creates underlying FileSystem (one from org.apache.hadoop) instance during it's construction. Single FileSystem instance is tied to a particular cluster (scheme + authority pair). In case we want to talk to another cluster, this fail due to FileSystem#checkPath.

 

This can be fixed by using FileSystem#get(java.net.URI, org.apache.hadoop.conf.Configuration) instead of FileSystem#newInstance(org.apache.hadoop.conf.Configuration){{}}

Issue Links

• links to: GitHub Pull Request #8923