Mahout
  1. Mahout
  2. MAHOUT-1005

Nuke Colt math functions that remain untested - The Matrix edition

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5, 0.6
    • Fix Version/s: 0.7
    • Component/s: Math
    • Labels:
      None

      Description

      We are down to very few references to Colt's matrix capabilities.

      I have implemented a new eigen-solver (with tests) and updated the Lanczos code to use it. I also have nuked all of the DoubleMatrix*d and related classes and consolidated several others.

      1. MAHOUT-1005.patch
        355 kB
        Ted Dunning

        Activity

        Hide
        Hudson added a comment -

        Integrated in Mahout-Quality #1463 (See https://builds.apache.org/job/Mahout-Quality/1463/)
        MAHOUT-1005 - Small updates trying for style points. (Revision 1334570)

        Result = SUCCESS
        tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334570
        Files :

        • /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java
        Show
        Hudson added a comment - Integrated in Mahout-Quality #1463 (See https://builds.apache.org/job/Mahout-Quality/1463/ ) MAHOUT-1005 - Small updates trying for style points. (Revision 1334570) Result = SUCCESS tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334570 Files : /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java
        Hide
        Mike Bria added a comment -

        Awesome dudes, thanks much. Looking forward to getting my Mahout on. Just got my 'Mahout In Action' this morning, so off I'll go soon!

        Show
        Mike Bria added a comment - Awesome dudes, thanks much. Looking forward to getting my Mahout on. Just got my 'Mahout In Action' this morning, so off I'll go soon!
        Hide
        Hudson added a comment -

        Integrated in Mahout-Quality #1462 (See https://builds.apache.org/job/Mahout-Quality/1462/)
        MAHOUT-1005 - Fixed error in transcription from JAMA, added test case to prove the case. (Revision 1334531)

        Result = SUCCESS
        tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334531
        Files :

        • /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java
        • /mahout/trunk/math/src/test/java/org/apache/mahout/math/solver/EigenDecompositionTest.java
        Show
        Hudson added a comment - Integrated in Mahout-Quality #1462 (See https://builds.apache.org/job/Mahout-Quality/1462/ ) MAHOUT-1005 - Fixed error in transcription from JAMA, added test case to prove the case. (Revision 1334531) Result = SUCCESS tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334531 Files : /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java /mahout/trunk/math/src/test/java/org/apache/mahout/math/solver/EigenDecompositionTest.java
        Hide
        Ted Dunning added a comment -

        Jake,

        Yes. The unit test used to pass. The problem was a transcription error on my part. I was testing the entire diagonal and should have been testing a shortening part of that diagonal.

        That problem is now tested and fixed. We should be good to go.

        Btw... relative to preserving the general form of the code, I could see from the Colt code that it was derived from the same source. That gave me a bit more confidence that the new version would behave similarly. These numerical codes live forever.

        Show
        Ted Dunning added a comment - Jake, Yes. The unit test used to pass. The problem was a transcription error on my part. I was testing the entire diagonal and should have been testing a shortening part of that diagonal. That problem is now tested and fixed. We should be good to go. Btw... relative to preserving the general form of the code, I could see from the Colt code that it was derived from the same source. That gave me a bit more confidence that the new version would behave similarly. These numerical codes live forever.
        Hide
        Ted Dunning added a comment -

        Lanczos problem fixed. It was due to a transcription error from the JAMA original.

        Show
        Ted Dunning added a comment - Lanczos problem fixed. It was due to a transcription error from the JAMA original.
        Hide
        Ted Dunning added a comment -

        Mike,

        Pestering is good.

        Fixing is better. I fixed the problem today and checked it in about 45 minutes ago. I just verified my fix on an independent machine with a clean checkout so you should be good to go.

        Show
        Ted Dunning added a comment - Mike, Pestering is good. Fixing is better. I fixed the problem today and checked it in about 45 minutes ago. I just verified my fix on an independent machine with a clean checkout so you should be good to go.
        Hide
        Mike Bria added a comment -

        Hi team,

        I don't mean to pester, but is there any progress on fixing this one?

        In the meantime - can you advise me on how I can get a clean build with this source? Today is my very first attempt at dl'ing/building/using Mahout (thanks a million in advance for your work!), and I'd like to see at least one build locally complete (even if this test is bypassed).

        Should I ignore the test in my source for now? Or maybe grab a previous tag from svn (if so, which one?)?

        Thanks very much,
        Mike

        ps// I also posted a note about this to the mailing list earlier today.
        pps/ is this kinda comment bad practice or frowned upon here?

        Show
        Mike Bria added a comment - Hi team, I don't mean to pester, but is there any progress on fixing this one? In the meantime - can you advise me on how I can get a clean build with this source? Today is my very first attempt at dl'ing/building/using Mahout (thanks a million in advance for your work!), and I'd like to see at least one build locally complete (even if this test is bypassed). Should I ignore the test in my source for now? Or maybe grab a previous tag from svn (if so, which one?)? Thanks very much, Mike ps// I also posted a note about this to the mailing list earlier today. pps/ is this kinda comment bad practice or frowned upon here?
        Hide
        Jake Mannix added a comment -

        So that test was passing before the change?

        It looks like this new solver thinks that the vectors are bigger than they should be? But then how could the DistributedLanczosSolverTest be passing in the other unit tests? Hmmmm...

        Show
        Jake Mannix added a comment - So that test was passing before the change? It looks like this new solver thinks that the vectors are bigger than they should be? But then how could the DistributedLanczosSolverTest be passing in the other unit tests? Hmmmm...
        Hide
        Hudson added a comment -

        Integrated in Mahout-Quality #1459 (See https://builds.apache.org/job/Mahout-Quality/1459/)
        MAHOUT-1005 - Added new implementation of EigenDecomposition based on JAMA code. Deleted old EigenvalueDecomposition from Colt. (Revision 1333667)

        Result = FAILURE
        tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333667
        Files :

        • /mahout/trunk/math/src/main/java/org/apache/mahout/math/decomposer/lanczos/LanczosSolver.java
        • /mahout/trunk/math/src/main/java/org/apache/mahout/math/matrix/linalg/EigenvalueDecomposition.java
        • /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java
        • /mahout/trunk/math/src/test/java/org/apache/mahout/math/decomposer/lanczos/TestLanczosSolver.java
        • /mahout/trunk/math/src/test/java/org/apache/mahout/math/solver/EigenDecompositionTest.java
        Show
        Hudson added a comment - Integrated in Mahout-Quality #1459 (See https://builds.apache.org/job/Mahout-Quality/1459/ ) MAHOUT-1005 - Added new implementation of EigenDecomposition based on JAMA code. Deleted old EigenvalueDecomposition from Colt. (Revision 1333667) Result = FAILURE tdunning : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333667 Files : /mahout/trunk/math/src/main/java/org/apache/mahout/math/decomposer/lanczos/LanczosSolver.java /mahout/trunk/math/src/main/java/org/apache/mahout/math/matrix/linalg/EigenvalueDecomposition.java /mahout/trunk/math/src/main/java/org/apache/mahout/math/solver/EigenDecomposition.java /mahout/trunk/math/src/test/java/org/apache/mahout/math/decomposer/lanczos/TestLanczosSolver.java /mahout/trunk/math/src/test/java/org/apache/mahout/math/solver/EigenDecompositionTest.java
        Hide
        Ted Dunning added a comment -

        Jake,

        Does this ring a bell?

        
            <error message="7" type="java.lang.ArrayIndexOutOfBoundsException">java.lang.ArrayIndexOutOfBoundsException: 7
                at org.apache.mahout.math.DenseVector.getQuick(DenseVector.java:110)
                at org.apache.mahout.math.solver.EigenDecomposition.tql2(EigenDecomposition.java:294)
                at org.apache.mahout.math.solver.EigenDecomposition.&lt;init&gt;(EigenDecomposition.java:78)
                at org.apache.mahout.math.solver.EigenDecomposition.&lt;init&gt;(EigenDecomposition.java:62)
                at org.apache.mahout.math.decomposer.lanczos.LanczosSolver.solve(LanczosSolver.java:141)
                at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:201)
                at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:124)
                at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver$DistributedLanczosSolverJob.run(DistributedLanczosSolver.java:284)
                at org.apache.mahout.math.hadoop.decomposer.TestDistributedLanczosSolverCLI.testDistributedLanczosSolverCLI(TestDistributedLanczosSolverCLI.java:76)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
                at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
                at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
                at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
                at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
                at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119)
                at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
        :
        
        Show
        Ted Dunning added a comment - Jake, Does this ring a bell? <error message= "7" type= "java.lang.ArrayIndexOutOfBoundsException" >java.lang.ArrayIndexOutOfBoundsException: 7 at org.apache.mahout.math.DenseVector.getQuick(DenseVector.java:110) at org.apache.mahout.math.solver.EigenDecomposition.tql2(EigenDecomposition.java:294) at org.apache.mahout.math.solver.EigenDecomposition.&lt;init&gt;(EigenDecomposition.java:78) at org.apache.mahout.math.solver.EigenDecomposition.&lt;init&gt;(EigenDecomposition.java:62) at org.apache.mahout.math.decomposer.lanczos.LanczosSolver.solve(LanczosSolver.java:141) at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:201) at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:124) at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver$DistributedLanczosSolverJob.run(DistributedLanczosSolver.java:284) at org.apache.mahout.math.hadoop.decomposer.TestDistributedLanczosSolverCLI.testDistributedLanczosSolverCLI(TestDistributedLanczosSolverCLI.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) :
        Hide
        Ted Dunning added a comment -

        I got a test failure in running full tests.

        Unfortunately, the test is in org.apache.mahout.math.hadoop.decomposer.TestDistributedLanczosSolverCLI and that test takes FOREVER.

        I will leave the commit in for a short while longer on the theory that the problem is small.

        Show
        Ted Dunning added a comment - I got a test failure in running full tests. Unfortunately, the test is in org.apache.mahout.math.hadoop.decomposer.TestDistributedLanczosSolverCLI and that test takes FOREVER. I will leave the commit in for a short while longer on the theory that the problem is small.
        Hide
        Ted Dunning added a comment -

        Will re-open if we see any obscure issues.

        Show
        Ted Dunning added a comment - Will re-open if we see any obscure issues.
        Show
        Ted Dunning added a comment - Work in progress visible at https://github.com/tdunning/mahout See especially the following commits: https://github.com/tdunning/mahout/commit/5750e1439d6afe75724c3f5d40ca0eca0e337b55 https://github.com/tdunning/mahout/commit/0c40be5525e90ec5066947cc172f6b51c52b00fd https://github.com/tdunning/mahout/commit/ddace9fafa3cf398abb479da9b0aa1976bd767df
        Hide
        Ted Dunning added a comment - - edited

        This patch may not apply without -p0

        Show
        Ted Dunning added a comment - - edited This patch may not apply without -p0

          People

          • Assignee:
            Ted Dunning
            Reporter:
            Ted Dunning
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development