Details
-
Sub-task
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Beam's FileSystem has a copy() command, however I can't find a good analog in Hadoop's FileSystem. https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html shows lots of copy to/from local files, but no "copy between these two arbitrary paths".
cc davor dhalperi@google.com did either of you have thoughts about this? I don't think that it makes sense to have beam stream data from one node just so it can write it back to another node. (it could be an extension method, but I'd want to make it obvious that it's the inefficient version of things)
My default answer here is to throw an unimplemented exception on copy or to remove it from the BFS interface altogether.