Uploaded image for project: 'Commons Math'
  1. Commons Math
  2. MATH-1068

KendallsCorrelation suffers from integer overflow for large arrays.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 3.3
    • 3.3
    • None

    Description

      For large array size (say, over 5,000), numPairs > 10 million.
      in line 258, (numPairs - tiedXPairs) * (numPairs - tiedYPairs) possibly > 100 billion, which will cause an integer overflow, resulting in a negative number, which will result in the end result in a NaN since the square-root of that number is calculated.
      This can easily be solved by changing line 163 to
      final long numPairs = ((long)n) * (n - 1) / 2; // to avoid overflow

      Attachments

        Activity

          People

            Unassigned Unassigned
            Terran-Ghost Gal Lalouche
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1m
                1m
                Remaining:
                Remaining Estimate - 1m
                1m
                Logged:
                Time Spent - Not Specified
                Not Specified