[HDFS-15592] DistCP fails with ViewHDFS and preserveEC options if the actual target path is non HDFS

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: viewfs, ViewHDFS
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 23/Sep/20 06:36
• Updated: 23/Sep/20 07:25

Description

When we configure target path mount point with Ozone (or any other fs), distcp will fail.

The reason is, if the src path having ec policy enabled, it will try to retain that properties.SO, in this case it is using DFS specific createFile API.
But here we have to ensure, tareget path can from non hdfs in ViewHDFS case. 

In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece of code.

 

if (preserveEC && sourceStatus.isErasureCoded()
 && sourceStatus instanceof HdfsFileStatus
 && targetFS instanceof DistributedFileSystem) {
 ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
}

 

Here it's just checking targetFs instanceof DistributedFileSystem, but in ViewHDFS case, fs will be DFS only but actual target can point to mounted fs. So, to handle this case, we should use resolvePath API and check the resolved target path scheme is dfs or or not.