Mahout
  1. Mahout
  2. MAHOUT-760

"org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest" test fails during install

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.5
    • Fix Version/s: 0.7
    • Labels:
    • Environment:

      Description

      mvn install core fails because of a single failed test - "org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest" with the following error (extracted from target/surefire-reports/org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest.txt)

      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.615 sec <<< FAILURE!
      testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest) Time elapsed: 6.587 sec <<< FAILURE!
      org.junit.ComparisonFailure: expected:<

      {[D=0, E=1, A=0, B=0, C]=1}

      > but was:<

      {[A=0, B=0, C=1, D=0, E]=1}

      >
      at org.junit.Assert.assertEquals(Assert.java:123)
      at org.junit.Assert.assertEquals(Assert.java:145)
      at org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest.testStartParallelFPGrowth(PFPGrowthTest.java:95)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
      at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
      at $Proxy0.invoke(Unknown Source)
      at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
      at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)

      Every other test in all the components succeed.

        Activity

        Hide
        Sean Owen added a comment -

        What JVM are you using, at what version of Mahout?
        Another user reported this exact problem, but only on version 0.5 (not HEAD) and only on the IBM JDK. Can you try HEAD?

        Show
        Sean Owen added a comment - What JVM are you using, at what version of Mahout? Another user reported this exact problem, but only on version 0.5 (not HEAD) and only on the IBM JDK. Can you try HEAD?
        Hide
        Chintamani added a comment -

        I have the same environment - Mahout 0.5 and IBM JDK.
        But I do not know about "HEAD". Is it a code-name for some Mahout version?

        Thanks

        Show
        Chintamani added a comment - I have the same environment - Mahout 0.5 and IBM JDK. But I do not know about "HEAD". Is it a code-name for some Mahout version? Thanks
        Hide
        Sean Owen added a comment -

        HEAD just means the latest version of the code from Subversion (source control) instead of the last released version. I am not sure what the issue is with the IBM JDK, but some change since 0.5 seems to have fixed it. I would be great if you could confirm.

        Show
        Sean Owen added a comment - HEAD just means the latest version of the code from Subversion (source control) instead of the last released version. I am not sure what the issue is with the IBM JDK, but some change since 0.5 seems to have fixed it. I would be great if you could confirm.
        Hide
        Chintamani added a comment -

        I got a new set of 132 failures now - I guess because of some SUN JAVA code (I am on IBM JDK). Here is the error.
        Results :

        Tests in error:
        testMapper(org.apache.mahout.df.mapreduce.partial.Step2MapperTest): com.sun.security.auth.UnixPrincipal
        testMapper(org.apache.mahout.df.mapreduce.partial.Step1MapperTest): org.apache.hadoop.security.UserGroupInformation (initialization failure)
        testProcessOutput(org.apache.mahout.df.mapreduce.partial.PartialBuilderTest): org.apache.hadoop.security.UserGroupInformation (initialization failure)
        ....

        Tests run: 650, Failures: 0, Errors: 132, Skipped: 0

        Here is what the core/target/surefire-reports/com.sun.security.auth.UnixPrincipal.txt says

        -------------------------------------------------------------------------------
        Test set: org.apache.mahout.df.mapreduce.partial.Step2MapperTest
        -------------------------------------------------------------------------------
        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.372 sec <<< FAILURE!
        testMapper(org.apache.mahout.df.mapreduce.partial.Step2MapperTest) Time elapsed: 1.28 sec <<< ERROR!
        java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
        at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:246)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:197)
        at org.apache.mahout.df.mapreduce.partial.Step2MapperTest.testMapper(Step2MapperTest.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
        at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
        at $Proxy0.invoke(Unknown Source)
        at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
        at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
        Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
        at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:76)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:396)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:653)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:358)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:619)
        ... 39 more

        I got the svn from here - http://subversion.apache.org/packages.html#suse (CollabNet Subversion Edge 2.0 (for Linux 64 bit))

        Then I ran "svn co http://svn.apache.org/repos/asf/mahout/trunk" before running "mvn install"

        Show
        Chintamani added a comment - I got a new set of 132 failures now - I guess because of some SUN JAVA code (I am on IBM JDK). Here is the error. Results : Tests in error: testMapper(org.apache.mahout.df.mapreduce.partial.Step2MapperTest): com.sun.security.auth.UnixPrincipal testMapper(org.apache.mahout.df.mapreduce.partial.Step1MapperTest): org.apache.hadoop.security.UserGroupInformation (initialization failure) testProcessOutput(org.apache.mahout.df.mapreduce.partial.PartialBuilderTest): org.apache.hadoop.security.UserGroupInformation (initialization failure) .... Tests run: 650, Failures: 0, Errors: 132, Skipped: 0 Here is what the core/target/surefire-reports/com.sun.security.auth.UnixPrincipal.txt says ------------------------------------------------------------------------------- Test set: org.apache.mahout.df.mapreduce.partial.Step2MapperTest ------------------------------------------------------------------------------- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.372 sec <<< FAILURE! testMapper(org.apache.mahout.df.mapreduce.partial.Step2MapperTest) Time elapsed: 1.28 sec <<< ERROR! java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:246) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:197) at org.apache.mahout.df.mapreduce.partial.Step2MapperTest.testMapper(Step2MapperTest.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103) at $Proxy0.invoke(Unknown Source) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69) Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:76) at java.net.URLClassLoader.findClass(URLClassLoader.java:396) at java.lang.ClassLoader.loadClass(ClassLoader.java:653) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:358) at java.lang.ClassLoader.loadClass(ClassLoader.java:619) ... 39 more I got the svn from here - http://subversion.apache.org/packages.html#suse (CollabNet Subversion Edge 2.0 (for Linux 64 bit)) Then I ran "svn co http://svn.apache.org/repos/asf/mahout/trunk " before running "mvn install"
        Hide
        Marc Millstone added a comment -

        I have posted a fix onto the wiki https://cwiki.apache.org/confluence/display/MAHOUT/BuildingMahout, the issue is that Mahout at the trunk depends on Hadoop 0.20.203.0, which has some proprietary Sun java stuff in there for authentication.

        The trick is to use Mahout Trunk + Hadoop 0.20.2.

        I think that in newer version of Hadoop, these sun dependencies are in fact fixed.

        Show
        Marc Millstone added a comment - I have posted a fix onto the wiki https://cwiki.apache.org/confluence/display/MAHOUT/BuildingMahout , the issue is that Mahout at the trunk depends on Hadoop 0.20.203.0, which has some proprietary Sun java stuff in there for authentication. The trick is to use Mahout Trunk + Hadoop 0.20.2. I think that in newer version of Hadoop, these sun dependencies are in fact fixed.
        Hide
        Chintamani added a comment -

        I changed pom.xml and it builds fine now.

        Show
        Chintamani added a comment - I changed pom.xml and it builds fine now.

          People

          • Assignee:
            Sean Owen
            Reporter:
            Chintamani
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development