[BEAM-2211] DataflowRunner (Java) rejects all but GCS paths for FileBasedSource/Sink

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: runner-dataflow
• Assignee: Dan Halperin
• Reporter: Dan Halperin
• Created: 08/May/17 16:37
• Updated: 24/Jul/20 20:13
• Resolved: 09/May/17 15:55

Description

FileBasedSource and Sink have switched in Beam to the FileSystems API from the the IOChannelUtils API, which means they now support HDFS and GCS and others.

However, the DataflowRunner still uses GcsPathValidator, which means it will likely currently disallow HDFS and other new FileSystem implementations.

Issue Links

• links to: GitHub Pull Request #2968
• links to: GitHub Pull Request #2972
• links to: GitHub Pull Request #2984
• links to: GitHub Pull Request #2991
• links to: GitHub Pull Request #3022
• links to: GitHub Pull Request #3048