Uploaded image for project: 'Commons Math'
  1. Commons Math
  2. MATH-1145

Integer overflows MannWhitneyUTest#mannWhitneyU

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3, 3.4
    • Fix Version/s: 3.4
    • Labels:
      None

      Description

      In the calculation of MannWhitneyUTest#mannWhitneyU there are two instances where the lengths of the input arrays are multiplied together. Because Array#length is an integer this means that the effective maximum size of your dataset until reaching overflow is Math.sqrt(Integer.MAX_VALUE).

      The following is a link to a diff, with a test the exposes the issue, and a fix (casting lengths up into doubles before multiplying).

      https://gist.github.com/aconbere/4fef56e5182e510aceb3

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aconbere Anders Conbere
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: