Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-683

LDA Vectorization

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5
    • Component/s: Clustering

      Description

      Currently the result of LDA clustering algorithm is a state which describes the probability of words, part of a corpus of documents, to belong to given topics. This probability is calculated for the whole corpus
      It is interesting, however, what is the average number of words of a given document that comes from a given topic. This information comes from the gamma vector in the LDA inference process. This vector can be used as representation of the given document for further clustering purposes (using algorithms like KMeans, Dirichlet, etc.). In this manner the dimensions of a document get reduced to the number of topics that is specified to the LDA clustering algorithm.
      With the proposed implementation from a corpus of documents described as vectors and from the last state of LDA inference process a set of vectors with reduced dimensions is produced (a vector per a document) which represent the set of documents

        Attachments

        1. MAHOUT-683.patch
          17 kB
          Vasil Vasilev

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vavasilev Vasil Vasilev
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: