Details

Type: New Feature

Status: Open

Priority: Major

Resolution: Unresolved

Affects Version/s: 0.5.0

Fix Version/s: None

Component/s: None

Labels:None

Environment:
UDF, written in Pig 0.5 contrib/

Tags:contrib udf variance standard deviation
Description
I've implemented a UDF in Pig 0.5 that implements Algebraic and calculates variance in a distributed manner, based on the AVG() builtin. It works by calculating the count, sum and sum of squares, as described here: http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
Is this a worthwhile contribution? Taking the square root of this value using the contrib SQRT() function gives Standard Deviation, which is missing from Pig.
I have attached an UDF that uses the Chan method. I used AVG as a guide, so I implemented Algebraic and Accumulator. If this is to be committed, I can add javadocs and test cases.