[BEAM-371] Backport HDFS IO enhancements from Scio

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 0.1.0-incubating
• Fix Version/s: 0.2.0-incubating
• Component/s: io-java-hadoop-file-system
• Assignee: Neville Li
• Reporter: Neville Li
• Created: 23/Jun/16 17:03
• Updated: 24/Jul/20 20:56
• Resolved: 02/Jul/16 02:55

Description

Right now there is a beam-sdks-java-io-hdfs module but only HDFSFileSource is implemented and there's a known issue with reading Avro files.
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/102

We at Spotify have implemented HDFS sinks, specialized source/sink for Avro and simple authentication and would like to port it back to Beam.

https://github.com/apache/incubator-beam/pull/485

Issue Links

• links to: GitHub Pull Request #485