[SPARK-1227] Diagnostics for Classification&Regression

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: MLlib
• Assignee: Martin Jaggi
• Reporter: Martin Jaggi
• Created: 06/Feb/14 03:20
• Updated: 07/Nov/16 18:03
• Resolved: 05/Nov/15 09:32

Description

Currently, the attained objective function is not computed (for efficiency reasons, as one evaluation requires one full pass through the data).

For diagnostics and comparing different algorithms, we should however provide this as a separate function (one MR).

Doing this requires the loss and regularizer functions themselves, not only their gradients (which are currently in the Gradient class). How about adding the new function directly on the corresponding models in classification/* and regression/* ? Any thoughts?

Issue Links

• relates to: SPARK-5256 Improving MLlib optimization APIs
• relates to: SPARK-18303 CLONE - Improving MLlib optimization APIs