[SPARK-8936] Hyperparameter estimation in LDA

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: ML
• Target Version/s: 1.5.0
• Assignee: Feynman Liang
• Reporter: Feynman Liang
• Shepherd: Joseph K. Bradley
• Created: 09/Jul/15 01:49
• Updated: 01/Aug/15 01:36
• Resolved: 01/Aug/15 01:36

Description

LDA currently supports inference of word distributions per topic (LDAModel.topicsMatrix) but no hyperparameter (e.g. document-topic Dirichlet parameter alpha, document-word Dirichlet parameter beta) point estimates are produced.

Hyperparameter estimation in Online LDA has been described by Blei (https://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf, equation 9) and should be supported in Spark's implementation.

Issue Links

• is required by: SPARK-5572 LDA improvement listing
• links to: [Github] Pull Request #7507 (feynmanliang)
• links to: [Github] Pull Request #7836 (feynmanliang)