[SPARK-6793] Implement perplexity for LDA

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.5.0
• Component/s: MLlib
• Target Version/s: 1.5.0
• Assignee: Feynman Liang
• Reporter: Joseph K. Bradley
• Shepherd: Joseph K. Bradley
• Created: 08/Apr/15 23:39
• Updated: 29/Jul/15 23:20
• Resolved: 29/Jul/15 23:20

Description

LDA should be able to compute perplexity. This JIRA is for computing it on the training dataset. See the linked JIRA for computing it on a new corpus: SPARK-5567

Issue Links

• is related to: SPARK-5567 Add prediction methods to LDA
• is required by: SPARK-5572 LDA improvement listing
• requires: SPARK-9440 LocalLDAModel should save docConcentration, topicConcentration, and gammaShape
• links to: [Github] Pull Request #7507 (feynmanliang)
• links to: [Github] Pull Request #7705 (feynmanliang)