[SPARK-11968] ALS recommend all methods spend most of time in GC - ASF JIRA

XML

Word

Printable

JSON

After adding recommendUsersForProducts and recommendProductsForUsers to ALS in spark-perf, I noticed that it takes much longer than ALS itself. Looking at the monitoring page, I can see it is spending about 8min doing GC for each 10min task. That sounds fixable. Looking at the implementation, there is clearly an opportunity to avoid extra allocations: https://github.com/apache/spark/blob/e6dd237463d2de8c506f0735dfdb3f43e8122513/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L283

is duplicated by

SPARK-20446 Optimize the process of MLLIB ALS recommendForAll

is related to

SPARK-3066 Support recommendAll in matrix factorization model

relates to

SPARK-13857 Feature parity for ALS ML with MLLIB

SPARK-20587 Improve performance of ML ALS recommendForAll

SPARK-10802 Let ALS recommend for subset of data

links to

[Github] Pull Request #9980 (rekhajoshm)

(2 links to)