Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4231

Add RankingMetrics to examples.MovieLensALS

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: Examples
    • Labels:
      None

      Description

      examples.MovieLensALS computes RMSE for movielens dataset but after addition of RankingMetrics and enhancements to ALS, it is critical to look at not only the RMSE but also measures like prec@k and MAP.

      In this JIRA we added RMSE and MAP computation for examples.MovieLensALS and also added a flag that takes an input whether user/product recommendation is being validated.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'debasish83' has created a pull request for this issue:
          https://github.com/apache/spark/pull/3098

          Show
          apachespark Apache Spark added a comment - User 'debasish83' has created a pull request for this issue: https://github.com/apache/spark/pull/3098
          Hide
          debasish83 Debasish Das added a comment -

          Shuo Xiang Xiangrui Meng Sean Owen

          I looked at Sean's implementation for MAP metric for recommendation engines and it computes a rank of predicted test set over all user/product predictions...I don't see how can I send the rank vector to the RankingMetrics API right now....

          http://cloudera.github.io/oryx/xref/com/cloudera/oryx/als/computation/local/ComputeMAP.html

          Right now for each user/product I send predictions and labels from test set to RankingMetrics API but there is no rank order defined...I retrieved the predictions using ALS.predict(userId, productId) API...

          Show
          debasish83 Debasish Das added a comment - Shuo Xiang Xiangrui Meng Sean Owen I looked at Sean's implementation for MAP metric for recommendation engines and it computes a rank of predicted test set over all user/product predictions...I don't see how can I send the rank vector to the RankingMetrics API right now.... http://cloudera.github.io/oryx/xref/com/cloudera/oryx/als/computation/local/ComputeMAP.html Right now for each user/product I send predictions and labels from test set to RankingMetrics API but there is no rank order defined...I retrieved the predictions using ALS.predict(userId, productId) API...
          Hide
          srowen Sean Owen added a comment -

          So this method basically computes where each test item would rank if you asked for a list of recommendations that ranks every single item. It's not necessarily efficient, but is simple. The reason I did it that way was to avoid recreating a lot of the recommender ranking logic. I don't think one has to define MAP this way – I effectively averaged over all k to the # of items.

          Yes I found the straightforward definition hard to implement at scale. I ended up opting to compute an approximation of AUC for recommender eval in this next version I'm working on: https://github.com/OryxProject/oryx/blob/master/oryx-ml-mllib/src/main/java/com/cloudera/oryx/ml/mllib/als/AUC.java#L106 Sorry for the hard-to-read Java 7; going to redo this in Java 8 soon. Basically you're just sampling random relevant/not-relevant pairs and comparing their scores. You might consider that.

          I dunno if it's worth bothering with a toy implementation in the examples. The example is already just to show Spark really not ALS.

          Show
          srowen Sean Owen added a comment - So this method basically computes where each test item would rank if you asked for a list of recommendations that ranks every single item. It's not necessarily efficient, but is simple. The reason I did it that way was to avoid recreating a lot of the recommender ranking logic. I don't think one has to define MAP this way – I effectively averaged over all k to the # of items. Yes I found the straightforward definition hard to implement at scale. I ended up opting to compute an approximation of AUC for recommender eval in this next version I'm working on: https://github.com/OryxProject/oryx/blob/master/oryx-ml-mllib/src/main/java/com/cloudera/oryx/ml/mllib/als/AUC.java#L106 Sorry for the hard-to-read Java 7; going to redo this in Java 8 soon. Basically you're just sampling random relevant/not-relevant pairs and comparing their scores. You might consider that. I dunno if it's worth bothering with a toy implementation in the examples. The example is already just to show Spark really not ALS.
          Hide
          debasish83 Debasish Das added a comment -

          Sean Owen I need a standard metric to report the ALS enhancements proposed in this PR: https://github.com/apache/spark/pull/2705 and more work that we are doing in this direction...

          The metric should be consistent for topic modeling using LSA as well as the PR can solve LSA/PLSA...I have not yet gone into measures like perplexity etc from LDA PRs as they are even more complicated but MAP, prec@k and ndcg@k are the measures people reported LDA results as well..

          Does the MAP definition I used in this PR looks correct to you ? Let me look into the AUC example...

          Show
          debasish83 Debasish Das added a comment - Sean Owen I need a standard metric to report the ALS enhancements proposed in this PR: https://github.com/apache/spark/pull/2705 and more work that we are doing in this direction... The metric should be consistent for topic modeling using LSA as well as the PR can solve LSA/PLSA...I have not yet gone into measures like perplexity etc from LDA PRs as they are even more complicated but MAP, prec@k and ndcg@k are the measures people reported LDA results as well.. Does the MAP definition I used in this PR looks correct to you ? Let me look into the AUC example...
          Hide
          srowen Sean Owen added a comment -

          Yes I'm mostly questioning implementing this in examples. The definition in RankingMetrics looks like the usual one to me – average from 1 to min(# recs, # relevant items). You could say the version you found above is 'extended' to look into the long tail (# recs = # items), although the long tail doesn't affect MAP much. Same definition, different limit.

          precision@k does not have the same question since there is one k value, not lots.
          AUC may not help you if you're comparing to other things for which you don't have AUC. It was a side comment mostly.
          (Anyway there is already an AUC implementation here which I am trying to see if I can use.)

          Show
          srowen Sean Owen added a comment - Yes I'm mostly questioning implementing this in examples. The definition in RankingMetrics looks like the usual one to me – average from 1 to min(# recs, # relevant items). You could say the version you found above is 'extended' to look into the long tail (# recs = # items), although the long tail doesn't affect MAP much. Same definition, different limit. precision@k does not have the same question since there is one k value, not lots. AUC may not help you if you're comparing to other things for which you don't have AUC. It was a side comment mostly. (Anyway there is already an AUC implementation here which I am trying to see if I can use.)
          Hide
          debasish83 Debasish Das added a comment -

          Sean Owen I added batch predict APIs for user and product recommendation in the PR (both batch and mini-batch APIs that takes a per user topK products)

          For the MAP calculation code I still have kept it in examples.MovieLensALS in this PR but I feel it should be part of MatrixFactorizationModel as well where client can send a RDD of userID and list of products they don't want to be recommended in the topK...I will add this API in MatrixFactorizationModel if it makes sense...

          Show
          debasish83 Debasish Das added a comment - Sean Owen I added batch predict APIs for user and product recommendation in the PR (both batch and mini-batch APIs that takes a per user topK products) For the MAP calculation code I still have kept it in examples.MovieLensALS in this PR but I feel it should be part of MatrixFactorizationModel as well where client can send a RDD of userID and list of products they don't want to be recommended in the topK...I will add this API in MatrixFactorizationModel if it makes sense...
          Hide
          apachespark Apache Spark added a comment -

          User 'debasish83' has created a pull request for this issue:
          https://github.com/apache/spark/pull/5869

          Show
          apachespark Apache Spark added a comment - User 'debasish83' has created a pull request for this issue: https://github.com/apache/spark/pull/5869
          Hide
          debasish83 Debasish Das added a comment -

          The code was not part of SPARK-3066 and so reopening...

          Show
          debasish83 Debasish Das added a comment - The code was not part of SPARK-3066 and so reopening...
          Hide
          srowen Sean Owen added a comment -

          This got reopened but looks like it really was resolved in https://github.com/apache/spark/pull/3098 for 1.4.0

          Show
          srowen Sean Owen added a comment - This got reopened but looks like it really was resolved in https://github.com/apache/spark/pull/3098 for 1.4.0

            People

            • Assignee:
              debasish83 Debasish Das
              Reporter:
              debasish83 Debasish Das
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development