[HADOOP-6365] distributed cache doesn't work with HDFS and another file system

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1
• Component/s: filecache

Environment

CentOS

• Assignee: Unassigned
• Reporter: Marc Colosimo
• Created: 05/Nov/09 20:42
• Updated: 29/Jul/14 19:58
• Resolved: 29/Jul/14 19:58

Description

This is a continuation of http://issues.apache.org/jira/browse/HADOOP-5635 (JIRA wouldn't let me edit that one). I found another issue with DistributedCache using something besides HDFS. In my case I have TWO active file systems, with HDFS being the default file system.

My fix includes two additional changes (from HADOOP-5635) to get it to work with another filesystem scheme (plus the changes from the original patch). I've tested this an it works with my code on HDFS with another filesystem. I have similar changes to mapreduce.filecacheTaskDistributedCacheManager and TrackerDistributedCacheManager (0.22.0).

Basically, URI.getPath() is called instead of URI.toString(). toString returns the scheme plus path which is important in finding the file to copy (getting the file system). Otherwise it searches the default file system (in this case HDFS) for the file.