[BEAM-2500] Add support for S3 as a Apache Beam FileSystem

• Type: New Feature
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: io-java-aws
• Assignee: Jacob Marble
• Reporter: Luke Cwik
• Created: 22/Jun/17 17:11
• Updated: 13/Nov/20 23:43
• Resolved: 27/Jan/18 20:37

Description

Note that this is for providing direct integration with S3 as an Apache Beam FileSystem.

There is already support for using the Hadoop S3 connector by depending on the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with a S3 configuration[3].

1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
2: https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
3: https://wiki.apache.org/hadoop/AmazonS3

Issue Links

• supercedes: BEAM-994 Support for S3 file source and sink
• supercedes: BEAM-284 Add AWS S3 support to FileBasedSink
• links to: GitHub Pull Request #257
• links to: GitHub Pull Request #4080
• links to: GitHub Pull Request #4080
• links to: GitHub Pull Request #4523