Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5571

LDA should handle text as well

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.3.0
    • None
    • MLlib

    Description

      Latent Dirichlet Allocation (LDA) currently operates only on vectors of word counts. It should also supporting training and prediction using text (Strings).

      This plan is sketched in the original LDA design doc.

      There should be:

      • runWithText() method which takes an RDD with a collection of Strings (bags of words). This will also index terms and compute a dictionary.
      • dictionary parameter for when LDA is run with word count vectors
      • prediction/feedback methods returning Strings (such as describeTopicsAsStrings, which is commented out in LDA currently)

      Attachments

        Issue Links

          is required by

          Umbrella - An overarching type made of sub-tasks SPARK-5572 LDA improvement listing

          • Major - Major loss of function.
          • Resolved
          requires

          New Feature - A new feature of the product, which has yet to be developed. SPARK-8169 Add StopWordsRemover as a transformer

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-9578 Stemmer feature transformer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              josephkb Joseph K. Bradley
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: