Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-418

Computing the pairwise similarities of the rows of a matrix

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.3
    • 0.4
    • classic
    • None

    Description

      In response to the wish from MAHOUT-362 and the latest discussion on the mailing list started by Kris Jack about computing a document similarity matrix, I tried to generalize the approach we're already using to compute the item-item-similarities for collaborative filtering.

      The job in the patch computes the pairwise similarity of the rows of a matrix in a distributed manner, is uses a SequenceFile<IntWritable,VectorWritable> as input and outputs such a file too. Custom similarity implementations can be supplied, I've already implemented tanimoto and cosine for demo and testing purposes. The algorithm is based on the one presented here: http://www.umiacs.umd.edu/~jimmylin/publications/Elsayed_etal_ACL2008_short.pdf

      I'd be glad if someone could verify the applicability of this approach by running it with a reasonably large input, I'm also worried that it might buffer to much data in certain steps.

      If you decide to include it in mahout, some more efforts and decisions (like more tests, more similarity measures, integration with DistributedRowMatrix) would need to be made, I guess.

      Attachments

        1. MAHOUT-418.patch
          44 kB
          Sebastian Schelter
        2. MAHOUT-418-2.patch
          44 kB
          Sebastian Schelter
        3. MAHOUT-418-3.patch
          217 kB
          Sebastian Schelter

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            ssc Sebastian Schelter
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment