Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5563

LDA with online variational inference

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.4.0
    • Component/s: MLlib
    • Labels:
      None
    • Target Version/s:

      Description

      Latent Dirichlet Allocation (LDA) parameters can be inferred using online variational inference, as in Hoffman, Blei and Bach. “Online Learning for Latent Dirichlet Allocation.” NIPS, 2010. This algorithm should be very efficient and should be able to handle much larger datasets than batch algorithms for LDA.

      This algorithm will also be important for supporting Streaming versions of LDA.

      The implementation will ideally use the same API as the existing LDA but use a different underlying optimizer.

      This will require hooking in to the existing mllib.optimization frameworks.

      This will require some discussion about whether batch versions of online variational inference should be supported, as well as what variational approximation should be used now or in the future.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yuhaoyan yuhao yang
                Reporter:
                josephkb Joseph K. Bradley
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: