[math] spearman rank cross correlation

• Key: MATH-136
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Phil Steitz
• Reporter: john gant
• Fix Version/s: 2.0

Environment

Operating System: other
Platform: Other

Description

added spearman rank correlation, along with unit test

Comments

Created an attachment (id=16171)
spearman rank cross correlation

[john gant]

Created an attachment (id=16172)
spearman rank cross correlation unit test

[Brent Worden]

I noticed you implementation does not take into account ties in the data. Ties
can effect the correlation coefficeint to a great degree.

A nice approach to address ties is to rank the data adjusting the ranks of tied
elements and then compute the Pearson's correlation coefficient on the rankings.

See http://web.uccs.edu/lbecker/SPSS/ctabs2.htm#5E.%20Spearman%20Correlation,%
20rs for a brief explaination.

If we get into adding more non-parametric routines, I for one would like to see
some general, ranking utililies such as taking an array of data and returning
the ranking array. The ranking could be driven by a tie ranking policy for
dealing with ties in the data. The default policy would be to use the mean
rank for ties. Other policies could be to omit the data, use the highest rank
or, use the lowest rank.

[Brent Worden]

Copied from the mailing list:
----- Original Message -----
From: "John Gant" <[e-mail]>
>
> Specifically testTwo() in
> http://issues.apache.org/bugzilla/attachment.cgi?id=16172 takes care
> of data with equal value (ie equal rank), is this the type of
> situation to which you are referring? Yes I agree, we should implement
> routines to sort in more diverse ways, but for right now I depend upon
> Arrays.sort() to perform the sorting.

Yes, and in this case the implementation is incorrectly computing the spearman
correlation as -0.1. But, according to R, the correlation is drastically
different:

> x <- c(2.0, 1.0, 3.0, 3.0, 5.0)
> y <- c(4.0, 4.0, 1.0, 2.0, 3.0)
> cor(x, y, method="spearman")
[1] -0.631579

Thus, I hold the implementation needs to change to correctly rank data with
ties.

[john gant]

Created an attachment (id=16184)
[math] ranking

interface for concrete implementations of ranking algorithms

[john gant]

Created an attachment (id=16185)
[math] ranking

Implementation of a generic ranking algorithm where ties get equivalent rank.

Need to implement the following ranking algorithms from R:

http://www.maths.lth.se/help/R/.R/library/base/html/rank.html

a. first
b. random
c. average

[john gant]

Created an attachment (id=16186)
[math] spearman rank cross correlation

Adapted previous code to use Ranking Algorithm implementations.

[john gant]

Created an attachment (id=16187)
[math] spearman rank cross correlation

Replaced faulty unit test, does not take into account all of R's ranking
implementations.

[john gant]

Created an attachment (id=16188)
[math] ranking

corresponding unit test

[Phil Steitz]

Implementation committed in r778085. User guide updated in 786940.

[Luc Maisonobe]

closing resolved issue for 2.0 release