[HIVE-3455] ANSI CORR(X,Y) is incorrect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.12.0
• Fix Version/s: 0.13.0
• Component/s: UDF
• Labels: patch

Release Note

the patch for the
src/ql/src/java/org/apache/hadoop/hive/ql/udf/generic

• Tags: correlation UDAF
• Assignee: Maxim Bolotin
• Reporter: Maxim Bolotin
• Created: 12/Sep/12 22:49
• Updated: 06/Dec/13 00:55
• Resolved: 06/Dec/13 00:55

Description

A simple test with 2 collinear vectors returns a wrong result.
The problem is the merge of variances, file:

http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java?revision=1157222&view=markup

lines:
347: myagg.xvar += xvarB + (xavgA - xavgB) * (xavgA - xavgB) * myagg.count;
348: myagg.yvar += yvarB + (yavgA - yavgB) * (yavgA - yavgB) * myagg.count;

the correct merge should be like this:
347: myagg.xvar += xvarB + (xavgA - xavgB) * (xavgA - xavgB) / myagg.count * nA * nB;
348: myagg.yvar += yvarB + (yavgA - yavgB) * (yavgA - yavgB) / myagg.count * nA * nB;

Issue Links

• is related to: HIVE-1549 Add ANSI SQL correlation aggregate function CORR(X,Y).