[PIG-1150] VAR() Variance UDF

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.5.0

Environment

UDF, written in Pig 0.5 contrib/

• Tags: contrib udf variance standard deviation
• Assignee: Dmitriy V. Ryaboy
• Reporter: Russell Jurney
• Created: 14/Dec/09 22:13
• Updated: 12/Apr/14 00:15

Description

I've implemented a UDF in Pig 0.5 that implements Algebraic and calculates variance in a distributed manner, based on the AVG() builtin. It works by calculating the count, sum and sum of squares, as described here: http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm

Is this a worthwhile contribution? Taking the square root of this value using the contrib SQRT() function gives Standard Deviation, which is missing from Pig.