[MAPREDUCE-7292] Use snapshot diff to build file listing when copying to blob storage

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: distcp
• Assignee: Unassigned
• Reporter: John Liu
• Created: 10/Aug/20 19:27
• Updated: 20/Aug/20 19:35

Description

The DistCp sync option should be extensible for copying to blob storage, which is not a distributed filesystem. Clients of DistCp could benefit from using the HDFS snapshot diff report to create the file listing in less time. A valid use case is to copy new files added to HDFS to a remote blob storage. The client ensures all new files are copied over but does not require the destination filesystem to be a distributed filesystem or have the previous snapshot.