[ARROW-7309] [Python] Support HDFS federation viewfs://

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.15.1
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: pull-request-available
• Assignee: Fabian Höring
• Reporter: Fabian Höring
• Created: 04/Dec/19 10:35
• Updated: 06/Jan/20 20:17
• Resolved: 06/Jan/20 20:16

Description

• Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
• libhdfs already supports injecting the scheme and will automatically resolve federation in
  fs = FileSystem#get(URI, conf, ugi)
• works with Hadoop 2/3

see:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637

Issue Links

• links to: GitHub Pull Request #5957