[HDFS-16911] Distcp with snapshot diff to support Ozone filesystem.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: distcp
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Sadanand Shenoy
• Reporter: Sadanand Shenoy
• Created: 07/Feb/23 15:29
• Updated: 28/Jan/24 07:19
• Resolved: 10/Apr/23 21:05

Description

Currently in DistcpSync i.e the step which applies the diff b/w 2 provided snapshots as arguments to the distcp job with -diff option, only DistributedFilesystem and WebHDFS filesytems are supported.

 

// currently we require both the source and the target file system are
// DistributedFileSystem or (S)WebHdfsFileSystem.
if (!(srcFs instanceof DistributedFileSystem
        || srcFs instanceof WebHdfsFileSystem)) {
  throw new IllegalArgumentException("Unsupported source file system: "
      + srcFs.getScheme() + "://. " +
      "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
}
if (!(tgtFs instanceof DistributedFileSystem
    || tgtFs instanceof WebHdfsFileSystem)) {
  throw new IllegalArgumentException("Unsupported target file system: "
      + tgtFs.getScheme() + "://. " +
      "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
}

As Ozone now supports snapshot feature after HDDS-6517, add support to use distcp with ozone snapshots.

 

Issue Links

• Dependent: HDDS-7916 Support for Distcp with Ozone Snapshots.
• is duplicated by: HADOOP-17719 DistCp with snapshot diff should support file systems supporting getSnapshotDiffReport
• is related to: HADOOP-17719 DistCp with snapshot diff should support file systems supporting getSnapshotDiffReport
• links to: GitHub Pull Request #5364
• links to: GitHub Pull Request #5840