Uploaded image for project: 'Commons Math'
  1. Commons Math
  2. MATH-238

MathUtils.gcd(u, v) fails when u and v both contain a high power of 2

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.0
    • None
    • None
    • None

    Description

      The test at the beginning of MathUtils.gcd(u, v) for arguments equal to zero fails when u and v contain high enough powers of 2 so that their product overflows to zero.

      assertEquals(3 * (1<<15), MathUtils.gcd(3 * (1<<20), 9 * (1<<15)));

      Fix: Replace the test at the start of MathUtils.gcd()

      if (u * v == 0) {

      by

      if (u == 0 || v == 0) {

      Attachments

        Activity

          People

            Unassigned Unassigned
            chsemrau Christian Semrau
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: