Mahout
  1. Mahout
  2. MAHOUT-393

Distributed item similarity functions

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4
    • Labels:
      None

      Description

      To complete the work started in MAHOUT-389, I've created a distributed version of any item similarity function that is currently already available in a non-distributed manner. An additional M/R job was necessary to compute the number of all users which is needed by some similarity functions (like LogLikelihoodSimilarity for example).

      There is still some optimization potential in the code as not every similarity function needs all information that is currently extracted (like the number of users e.g.), but the optimization would have made the code much less readable so I did not do any work on that.

      I hope you consider this a useful addition.

      1. MAHOUT-393.patch
        54 kB
        Sebastian Schelter

        Activity

        Hide
        Sebastian Schelter added a comment -

        org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityTest.testCompleteJob() explicitly tested for the number of users and it worked with your changes, so it's good as it is it seems

        Show
        Sebastian Schelter added a comment - org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityTest.testCompleteJob() explicitly tested for the number of users and it worked with your changes, so it's good as it is it seems
        Hide
        Sean Owen added a comment -

        Unless, I missed something, and the unit tests don't manage to catch it, yeah I think this is important. The way it was defined, the objects had an ordering but all were equal. So compareTo() would return nonzero for objects that are equals() and that could cause problems if not now then some day. (It may happen that the value of equals() is never used). Anyway, all set here it seems.

        Show
        Sean Owen added a comment - Unless, I missed something, and the unit tests don't manage to catch it, yeah I think this is important. The way it was defined, the objects had an ordering but all were equal. So compareTo() would return nonzero for objects that are equals() and that could cause problems if not now then some day. (It may happen that the value of equals() is never used). Anyway, all set here it seems.
        Hide
        Sebastian Schelter added a comment -

        I thought those definitions of equals() and hashCode() were necessary for the Secondary Sort to work, but obviously they aren't

        Show
        Sebastian Schelter added a comment - I thought those definitions of equals() and hashCode() were necessary for the Secondary Sort to work, but obviously they aren't
        Hide
        Sean Owen added a comment -

        Done, I committed with only two substantive tweaks:

        • I had switched over to VLongWritable from LongWritable. Most IDs used don't really need nearly 8 bytes, so variable-length coding saves a lot.
        • CountUsersKeyWritable didn't define equals() and hashCode() non-trivially, and was inconsistent with compareTo(). Do I miss something about this?
        Show
        Sean Owen added a comment - Done, I committed with only two substantive tweaks: I had switched over to VLongWritable from LongWritable. Most IDs used don't really need nearly 8 bytes, so variable-length coding saves a lot. CountUsersKeyWritable didn't define equals() and hashCode() non-trivially, and was inconsistent with compareTo(). Do I miss something about this?

          People

          • Assignee:
            Sean Owen
            Reporter:
            Sebastian Schelter
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development