[MAHOUT-468] Performance of RowSimilarityJob is not good

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.4
• Assignee: Unassigned
• Reporter: Han Hui Wen
• Created: 12/Aug/10 14:07
• Updated: 31/Jan/24 22:11
• Resolved: 14/Aug/10 18:35

Description

I have done a test ,

Preferences records: 680,194
distinct users: 23,246
distinct items:437,569
SIMILARITY_CLASS_NAME=SIMILARITY_COOCCURRENCE

maybePruneItemUserMatrixPath:16.50M
weights:13.80M
pairwiseSimilarity:18.81G
Job RowSimilarityJob-RowWeightMapper-WeightedOccurrencesPerColumnReducer:used 32 sec
Job RowSimilarityJob-CooccurrencesMapper-SimilarityReducer:used 4.30 hours

I think the reason may be following:
1) We used SequenceFileOutputFormat,it cause job can only be run by n ( n= Hadoop node counts ) mappers or reducers concurrently.
2) We stored redundant info.

for example :

the output of CooccurrencesMapper: (ItemIndexA,similarity),(ItemIndexA,ItemIndexB,similarity)

3) Some frequently used code
https://issues.apache.org/jira/browse/MAHOUT-467

4) allocate many local variable in loop (need confirm )

In Class DistributedUncenteredZeroAssumingCosineVectorSimilarity

@Override
public double weight(Vector v) {
double length = 0.0;

Iterator<Element> elemIterator = v.iterateNonZero();

while (elemIterator.hasNext())

{ double value = elemIterator.next().get(); //this one length += value * value; }

return Math.sqrt(length);
}

5) Maybe we need control the size of cooccurrences

Issue Links

• duplicates: MAHOUT-460 Add "maxPreferencesPerItemConsidered" option to o.a.m.cf.taste.hadoop.similarity.item.ItemSimilarityJob