Details
-
New Feature
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
Right now, FileBasedSource and FileBasedSink communication is mediated by IOChannelFactory. There are a number of issues:
- Global configuration – e.g., all 'gs://' URIs use the same credentials. This should be per-source/per-sink/etc.
- Supported APIs – currently IOChannelFactory is in the "non-public API" util package and subject to change. We need users to be able to add new backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be broken.
- Per-backend features: e.g., creating buckets in GCS/s3, setting expiration time, etc.
Updates:
Design docs posted on dev@ list:
Part 1: IOChannelFactory Redesign:
https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
Part 2: Configurable BeamFileSystem:
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs
Attachments
Issue Links
1.
|
IOChannelFactory: supports for recursively remove(). |
|
Open | Unassigned |
2.
|
Windows OS compatibilities |
|
Open | Unassigned |
3.
|
FileBasedSource: replace SeekableByteChannel with open(spec, startingPosition) |
|
Open | Unassigned |
4.
|
FileBaseSink: Handle output to locations such as ".", "~", or maybe ".." |
|
Open | Unassigned |
5.
|
URI Java library unexpected behavior. |
|
Open | Unassigned |