[SPARK-9836] Provide R-like summary statistics for ordinary least squares via normal equation solver

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: ML
• Target Version/s: 1.6.0
• Assignee: Yanbo Liang
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 11/Aug/15 19:34
• Updated: 03/Nov/15 16:29
• Resolved: 03/Nov/15 16:29

Description

In R, model fitting comes with summary statistics. We can provide most of those via normal equation solver (SPARK-9834). If some statistics requires additional passes to the dataset, we can expose an option to let users select desired statistics before model fitting.

> summary(model)

Call:
glm(formula = Sepal.Length ~ Sepal.Width + Species, data = iris)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.30711  -0.25713  -0.05325   0.19542   1.41253  

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)         2.2514     0.3698   6.089 9.57e-09 ***
Sepal.Width         0.8036     0.1063   7.557 4.19e-12 ***
Speciesversicolor   1.4587     0.1121  13.012  < 2e-16 ***
Speciesvirginica    1.9468     0.1000  19.465  < 2e-16 ***
---
Signif. codes:  
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 0.1918059)

    Null deviance: 102.168  on 149  degrees of freedom
Residual deviance:  28.004  on 146  degrees of freedom
AIC: 183.94

Number of Fisher Scoring iterations: 2

Issue Links

• blocks: SPARK-9837 Provide R-like summary statistics for GLMs via iteratively reweighted least squares
• is blocked by: SPARK-10668 Use WeightedLeastSquares in LinearRegression with L2 regularization if the number of features is small
• is blocked by: SPARK-9834 Normal equation solver for ordinary least squares
• links to: [Github] Pull Request #9413 (yanboliang)