Mahout
  1. Mahout
  2. MAHOUT-818

Canopy Emits Too Many Trivial Clusters

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5
    • Fix Version/s: 0.6
    • Component/s: Clustering
    • Labels:
      None

      Description

      Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, cf (-clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.

      1. vector.tar.gz
        1.89 MB
        beneo
      2. MAHOUT-818.patch
        30 kB
        Jeff Eastman

        Activity

        Jeff Eastman created issue -
        Hide
        Jeff Eastman added a comment -

        This patch implements a new Canopy CLI argument which limits the minimum size of output canopies. Unit tests have been updated and all run.

        Show
        Jeff Eastman added a comment - This patch implements a new Canopy CLI argument which limits the minimum size of output canopies. Unit tests have been updated and all run.
        Jeff Eastman made changes -
        Field Original Value New Value
        Attachment MAHOUT-818.patch [ 12496576 ]
        Hide
        Hudson added a comment -

        Integrated in Mahout-Quality #1066 (See https://builds.apache.org/job/Mahout-Quality/1066/)
        MAHOUT-818: Patch to filter canopies based upon numPoints. All tests run

        jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176603
        Files :

        • /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyConfigKeys.java
        • /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyDriver.java
        • /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyMapper.java
        • /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyReducer.java
        • /mahout/trunk/core/src/main/java/org/apache/mahout/common/commandline/DefaultOptionCreator.java
        • /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/canopy/TestCanopyCreation.java
        • /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayCanopy.java
        • /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayFuzzyKMeans.java
        Show
        Hudson added a comment - Integrated in Mahout-Quality #1066 (See https://builds.apache.org/job/Mahout-Quality/1066/ ) MAHOUT-818 : Patch to filter canopies based upon numPoints. All tests run jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176603 Files : /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyConfigKeys.java /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyDriver.java /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyMapper.java /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyReducer.java /mahout/trunk/core/src/main/java/org/apache/mahout/common/commandline/DefaultOptionCreator.java /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/canopy/TestCanopyCreation.java /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayCanopy.java /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayFuzzyKMeans.java
        Hide
        Jeff Eastman added a comment -

        Patch committed to trunk

        Show
        Jeff Eastman added a comment - Patch committed to trunk
        Jeff Eastman made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        beneo added a comment -

        this path make the clustering result bad

        i add the vector i used for the test, there is 5000 vectors

        canopy t1 = 0.032 t2 = 0.008
        kmeans x = 10 cd = 0.1

        if clusterFilter = 0
        the result is awesome
        if clusterFilter = 1
        OMG.....

        However, the clusterFilter make the canopy very faster, it's important for the 3Billions vector canopy..

        Show
        beneo added a comment - this path make the clustering result bad i add the vector i used for the test, there is 5000 vectors canopy t1 = 0.032 t2 = 0.008 kmeans x = 10 cd = 0.1 if clusterFilter = 0 the result is awesome if clusterFilter = 1 OMG..... However, the clusterFilter make the canopy very faster, it's important for the 3Billions vector canopy..
        Hide
        beneo added a comment -

        this is the vectors for my test.

        you can use this to watch the different result with clusterFilter or not

        the canopy t1=0.032, t2=0.008
        the kmeans x=10 cd=0.1

        Show
        beneo added a comment - this is the vectors for my test. you can use this to watch the different result with clusterFilter or not the canopy t1=0.032, t2=0.008 the kmeans x=10 cd=0.1
        beneo made changes -
        Attachment vector.tar.gz [ 12498732 ]
        Sean Owen made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jeff Eastman
            Reporter:
            Jeff Eastman
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development