Details
-
New Feature
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
Right now, FileBasedSource and FileBasedSink communication is mediated by IOChannelFactory. There are a number of issues:
- Global configuration – e.g., all 'gs://' URIs use the same credentials. This should be per-source/per-sink/etc.
- Supported APIs – currently IOChannelFactory is in the "non-public API" util package and subject to change. We need users to be able to add new backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be broken.
- Per-backend features: e.g., creating buckets in GCS/s3, setting expiration time, etc.
Updates:
Design docs posted on dev@ list:
Part 1: IOChannelFactory Redesign:
https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
Part 2: Configurable BeamFileSystem:
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs
Attachments
Issue Links
1.
|
Cleanup FileBasedSink to prepare IOChannelFactory redesign. |
|
Resolved | Pei He |
2.
|
Dynamically load and register IOChannelFactory |
|
Resolved | Pei He |
3.
|
FileBasedSink: remove unused code of TemporaryFileRetention |
|
Resolved | Pei He |
4.
|
FileBasedSink: removed unused code of TemporaryFileRetention |
|
Resolved | Pei He |
5.
|
IOChannelFactory: supports for recursively remove(). |
|
Open | Unassigned |
6.
|
Windows OS compatibilities |
|
Open | Unassigned |
7.
|
FileBasedSource: replace SeekableByteChannel with open(spec, startingPosition) |
|
Open | Unassigned |
8.
|
FileBaseSink: Handle output to locations such as ".", "~", or maybe ".." |
|
Open | Unassigned |
9.
|
URI Java library unexpected behavior. |
|
Open | Unassigned |
10.
|
Support file scheme |
|
Resolved | Unassigned |
11.
|
Update hadoop-file-system README.md |
|
Resolved | Ahmet Altay |