[SPARK-17151] Decide how to handle inferring number of classes in Multinomial logistic regression

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: ML, MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Seth Hendrickson
• Created: 19/Aug/16 05:29
• Updated: 21/May/19 04:37
• Resolved: 21/May/19 04:37

Description

This JIRA is to discuss how the number of label classes should be inferred in multinomial logistic regression. Currently, MLOR checks the dataframe metadata and if the number of classes is not specified then it uses the maximum value seen in the label column. If the labels are not properly indexed, then this can cause a large number of zero coefficients and potentially produce instabilities in model training.