[REEF-1892] HDFS File Copy only uses local HDFS

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.17
• Component/s: REEF.NET IO
• Assignee: Unassigned
• Reporter: Rogan Carr
• Created: 24/Sep/17 05:55
• Updated: 14/Oct/17 16:09

Description

In REEF-1827 [1], the URI used to specify remote and local files were changed to use the "AbsolutePath". [2]
This means that a file specified as "hdfs://my/file" becomes "/my/file" and the hdfs:// is assumed by the `dfs` command.

This is fine if you are using vanilla HDFS, but for cases like Blob Storage in Azure, there is a special prefix, `wasb://` that is used instead of `hdfs://`. This means that the AbsolutePath method trims off the wasb, and this Copy() function instead attempts to download the file from the local HDFS instead of WASB.

We need to revisit this issue and keep the full path for copies while also keeping proper casing in the path.