[HDFS-13916] Distcp SnapshotDiff to support WebHDFS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1, 3.1.1
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: distcp, webhdfs
• Labels: easyfix newbie patch pull-request-available
• Target Version/s: 3.4.0, 3.3.2
• Hadoop Flags: Reviewed
• Flags: Patch
• Assignee: Xun REN
• Reporter: Xun REN
• Created: 13/Sep/18 19:13
• Updated: 20/Jul/23 16:41
• Resolved: 26/Jun/21 21:07

Description

ljain has worked on the JIRA: HDFS-13052 to provide the possibility to make DistCP of SnapshotDiff with WebHDFSFileSystem. However, in the patch, there is no modification for the real java class which is used by launching the command "hadoop distcp ..."

 

You can check in the latest version here:

https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java#L96-L100

In the method "preSyncCheck" of the class "DistCpSync", we still check if the file system is DFS. 

So I propose to change the class DistCpSync in order to take into consideration what was committed by Lokesh Jain.

Issue Links

• is related to: HADOOP-15777 Add support of getSnapshotDiff for all the FS inherit from FileSystem
• is related to: HDFS-13922 Provide support for "hdfs snapshotDiff webhdfs://my_path"
• relates to: HADOOP-17719 DistCp with snapshot diff should support file systems supporting getSnapshotDiffReport
• relates to: HDFS-16091 WebHDFS should support getSnapshotDiffReportListing
• relates to: HDFS-13052 WebHDFS: Add support for snasphot diff
• links to: GitHub Pull Request #5839