[SPARK-10597] MultivariateOnlineSummarizer for weighted instances

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.5.0
• Component/s: MLlib
• Labels: bulk-closed
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 14/Sep/15 20:15
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

MultivariateOnlineSummarizer for weighted instances is implemented as private API for SPARK-7685.

In SPARK-7685, the online numerical stable version of unbiased estimation of variance defined by the reliability weights: [https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Reliability_weights] is implemented, but we would like to make it as public api since there are different use-cases.

Currently, `count` will return the actual number of instances, and ignores instance weights, but `numNonzeros` will return the weighted # of nonzeros.

We need to decide the behavior of them before making it public.