Mahout needs to respect the file system type when getting a FileSystem for an input or output path

• Key: MAHOUT-118
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Jeff Eastman
• Reporter: Stephen Green
• Component/s: Classification, Clustering, Collaborative Filtering, Genetic Algorithms, Matrix
• Affects Version/s: 0.1, 0.2
• Fix Version/s: 0.1, 0.2

Description

All of the uses of org.apache.hadoop.fs.FileSystem.get use the single argument version that takes a job configuration. This will always return the default file system type (which is usually HDFS), rather than using the file system type used in the URIs for the input or output paths. This is particularly a problem on Amazon's Elastic MapReduce where the input and output data typically reside in a org.apache.hadoop.fs.s3native.NativeS3FileSystem.

Comments

This is a patch for mahout-trunk that fixes all of the occurrences (except one) of FileSystem.get to use the two argument version with an appropriate Path as a first argument.

The only remaining instance is in org.apache.mahout.ga.watchmaker.MahoutEvaluator at line 64, where there's no obvious Path available to provide the first argument. Getting this one working will probably require refactoring that's beyond the scope of my Mahout understanding.

[Jeff Eastman]

committed in r765769

[Deneche A. Hakim]

The only remaining instance is in org.apache.mahout.ga.watchmaker.MahoutEvaluator at line 64, where there's no obvious Path available to provide the first argument. Getting this one working will probably require refactoring that's beyond the scope of my Mahout understanding.

In this particular case MahoutEvaluator creates the input and output paths automatically, I think the default file system should do

[Jeff Eastman]

Given Deneche's comment above I'm going to mark this issue fixed. Thanks Steve.