[SPARK-5567] Add prediction methods to LDA

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.5.0
• Component/s: MLlib
• Target Version/s: 1.5.0
• Assignee: Feynman Liang Assign to me
• Reporter: Joseph K. Bradley
• Created: 03/Feb/15 18:55
• Updated: 29/Jul/16 01:40
• Resolved: 30/Jul/15 20:18

Description

LDA currently supports prediction on the training set. E.g., you can call logLikelihood and topicDistributions to get that info for the training data. However, it should support the same functionality for new (test) documents.

This will require inference but should be able to use the same code, with a few modification to keep the inferred topics fixed.

Note: The API for these methods is already in the code but is commented out.

Issue Links

• is depended upon by: SPARK-16786 LDA topic distributions for new documents in PySpark
• is related to: SPARK-8696 Streaming API for Online LDA
• is required by: SPARK-5572 LDA improvement listing
• relates to: SPARK-6793 Implement perplexity for LDA
• links to: [Github] Pull Request #7507 (feynmanliang)
• links to: [Github] Pull Request #7760 (feynmanliang)