Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1330

Unable to do K-means clustering on Reuters dataset

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.8
    • Fix Version/s: 0.9
    • Component/s: Clustering
    • Labels:
      None
    • Environment:

      Linux

      Description

      The attached code uses the Mahout API to do k-means clustering on the Reuters dataset and generates the initial centroids using the canopy algorithm. The parameters are exactly the same as the ones in the Scala example presented in the following link:

      http://sujitpal.blogspot.com/2012/09/learning-mahout-clustering.html

      The code compiles without an error, but the K-means algorithm cannot initiate because the initial centroids are not being generated. This in turn is due to the fact that the TF-IDF vectors are not being generated.

      Considering that this code compiles and is based on earlier Scala code that worked, it is suggestive that there is a bug in the Mahout source code that may need fixing. I thought I should bring it to your attention.

      I have attached the source code, the included JAR files and the shell script (called test-kmeans-clustering-reuters-java-api.sh) to compile and run the code. The output of the shell script is located in NewsKMeansClustering-output.txt. Please note that you may need to change the path (see environmental variable JARPATH) to the JAR files in the shell script based on where you put the JARs. I also attached the output of clusterdump utility in the form of .txt files for the intermediate outputs of my code such as the TF vectors and TF-IDF vectors (see tf-vectors.txt, tfidf-vectors.txt, df-count.txt and frequency-file.txt).

        Attachments

        1. df-count.txt
          702 kB
          Karthik Prakhya
        2. frequency-file.txt
          702 kB
          Karthik Prakhya
        3. MyAnalyzer.java
          2 kB
          Karthik Prakhya
        4. NewsKMeansClustering.java
          6 kB
          Karthik Prakhya
        5. NewsKMeansClustering-output.txt
          51 kB
          Karthik Prakhya
        6. reuters-seqfiles.zipx
          9.58 MB
          Karthik Prakhya
        7. test-kmeans-clustering-reuters-java-api.sh
          0.7 kB
          Karthik Prakhya
        8. tfidf-vectors.txt
          0.4 kB
          Karthik Prakhya

          Activity

            People

            • Assignee:
              smarthi Suneel Marthi
              Reporter:
              kprakhya Karthik Prakhya
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: