[SPARK-17824] QR solver for WeightedLeastSquares

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Yanbo Liang
• Reporter: Yanbo Liang
• Created: 07/Oct/16 08:07
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

Cholesky decomposition is unstable (for near-singular and rank deficient matrices) and only works on positive definite matrices which can not be guaranteed in all cases, it was often used when matrix A is very large and sparse due to faster calculation. QR decomposition has better numerical properties than Cholesky and can works on matrices which are not positive definite. Spark MLlib WeightedLeastSquares use Cholesky decomposition to solve normal equation currently, we should also support or move to QR solver for better stability. I'm preparing to send a PR.

cc dbtsai sethah

Issue Links

• depends upon: SPARK-17748 One-pass algorithm for linear regression with L1 and elastic-net penalties