[BEAM-2138] Support atomic rename within FileSystem to replace inefficient Hadoop copy

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-hadoop-file-system
• Assignee: Unassigned
• Reporter: Luke Cwik
• Created: 02/May/17 14:18
• Updated: 03/Jun/22 17:18

Description

Hadoop copy operation is inefficient since it needs to stream the entirety of the resource through the machine performing the copy. Hadoop file system implementations do support an efficient rename.

Apache Beam sinks rely on being able to rename files atomically which is currently done by using FileSystem copy + delete.