[MAHOUT-1286] Memory-efficient DataModel, supporting fast online updates and element-wise iteration

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9
• Fix Version/s: 0.9
• Labels: collaborative-filtering datamodel patch recommender
• Assignee: Suneel Marthi
• Reporter: Peng Cheng
• Created: 17/Jul/13 21:57
• Updated: 31/Jan/24 22:11
• Resolved: 03/Dec/13 03:44

Description

Most DataModel implementation in current CF component use hash map to enable fast 2d indexing and update. This is not memory-efficient for big data set. e.g. Netflix prize dataset takes 11G heap space as a FileDataModel.

Improved implementation of DataModel should use more compact data structure (like arrays), this can trade a little of time complexity in 2d indexing for vast improvement in memory efficiency. In addition, any online recommender or online-to-batch converted recommender will not be affected by this in training process.

Issue Links

• blocks: MAHOUT-1274 SGD-based Online SVD recommender