[BEAM-2005] Add a Hadoop FileSystem implementation of Beam's FileSystem

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: io-java-hadoop-file-system
• Assignee: Luke Cwik
• Reporter: Stephen Sisk
• Created: 18/Apr/17 21:10
• Updated: 16/May/20 13:39
• Resolved: 02/May/17 04:38

Description

Beam's FileSystem creates an abstraction for reading from files in many different places.

We should add a Hadoop FileSystem implementation (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html) - that would enable us to read from any file system that implements FileSystem (including HDFS, azure, s3, etc..)

I'm investigating this now.

Issue Links

• is depended upon by: BEAM-2016 Delete HDFSFileSource/Sink
• links to: GitHub Pull Request #2757
• links to: GitHub Pull Request #2763
• links to: GitHub Pull Request #2772
• links to: GitHub Pull Request #2776
• links to: GitHub Pull Request #2777

Sub-Tasks

• Implement beam FileSystem's copy(): Resolved
• Hadoop FileSystem needs to receive Hadoop Configuration: Resolved
• Implement delete: Resolved
• Implement ResourceIds for HadoopFileSystem: Resolved
• Implement match for HadoopFileSystem: Resolved