[MAHOUT-1019] VectorDistanceSimilarityJob

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8
• Fix Version/s: 0.8
• Component/s: classic
• Labels: VectorDistanceSimilarityJob distance vector

Environment

all

• Assignee: Unassigned
• Reporter: Timothy Potter
• Created: 21/May/12 20:18
• Updated: 31/Jan/24 22:16
• Resolved: 11/Mar/13 11:15

Description

The VectorDistanceSimilarityJob is a fantastic tool, but poses the risk of creating terabytes of output of dubious value. For example, I have ~10K seed vectors and millions of vectors to compute the similarity between so I would like to add an optional parameter to this job to specify a maximum distance threshold that prevents any distances above this value from being written to the output. The default would be 1.0d so no filtering is applied which ensures backwards compatibility, but if supplied, only rows where the distance is less than the threshold would be output from the mapper. This can help reduce the storage requirements of the output immensely. Probably name the parameter something like: noOutputIfDistanceGreaterThan