[MAHOUT-1707] Spark-itemsimilarity uses too much memory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.1

Environment

Spark

• Assignee: Pat Ferrel
• Reporter: Pat Ferrel
• Created: 13/May/15 17:23
• Updated: 31/Jan/24 22:16
• Resolved: 22/May/15 19:58

Description

java.lang.OutOfMemoryError: Java heap space

The code has an unnecessary .collect(), forcing all interaction data into memory of the client/driver. Increasing the executor memory will not help with this.

remove this line and rebuild Mahout.
https://github.com/apache/mahout/blob/mahout-0.10.x/spark/src/main/scala/org/apache/mahout/drivers/TextDelimitedReaderWriter.scala#L157

The errant line reads:

interactions.collect()

This forces the user action data into memory, a bad thing for memory consumption. Removing it should allow for better Spark memory management.