Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-180

port Hadoop-ified Lanczos SVD implementation from decomposer

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.2
    • 0.3
    • Math
    • None

    Description

      I wrote up a hadoop version of the Lanczos algorithm for performing SVD on sparse matrices available at http://decomposer.googlecode.com/, which is Apache-licensed, and I'm willing to donate it. I'll have to port over the implementation to use Mahout vectors, or else add in these vectors as well.

      Current issues with the decomposer implementation include: if your matrix is really big, you need to re-normalize before decomposition: find the largest eigenvalue first, and divide all your rows by that value, then decompose, or else you'll blow over Double.MAX_VALUE once you've run too many iterations (the L^2 norm of intermediate vectors grows roughly as (largest-eigenvalue)^(num-eigenvalues-found-so-far), so losing precision on the lower end is better than blowing over MAX_VALUE). When this is ported to Mahout, we should add in the capability to do this automatically (run a couple iterations to find the largest eigenvalue, save that, then iterate while scaling vectors by 1/max_eigenvalue).

      Attachments

        1. MAHOUT-180.patch
          52 kB
          Jake Mannix
        2. MAHOUT-180.patch
          66 kB
          Jake Mannix
        3. MAHOUT-180.patch
          48 kB
          Jake Mannix
        4. MAHOUT-180.patch
          63 kB
          Jake Mannix
        5. MAHOUT-180.patch
          66 kB
          Jake Mannix

        Issue Links

          Activity

            People

              jake.mannix Jake Mannix
              jake.mannix Jake Mannix
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: