[SPARK-4611] Implement the efficient vector norm

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: MLlib
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 26/Nov/14 01:05
• Updated: 02/Dec/14 03:41
• Resolved: 02/Dec/14 03:41

Description

The vector norm in breeze is implemented by `activeIterator` which is known to be very slow. In this PR, an efficient vector norm is implemented, and with this API, `Normalizer` has huge performance gain.

Here is the benchmark against mnist8m dataset.

Before

DenseVector: 68.25secs
SparseVector: 17.01secs

With this PR

DenseVector: 12.71secs
SparseVector: 2.73secs

Issue Links

• links to: [Github] Pull Request #3462 (dbtsai)