Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28519

Tests failed on aarch64 due the value of math.log and power function is different

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
    • Docs Text:
      Hide
      The result of `java.lang.Math`'s `log`, `log1p`, `exp`, `expm1`, and `pow` may vary across platforms. In Spark 3.0, the result of the equivalent SQL functions (including related SQL functions like `LOG10`) return values consistent with `java.lang.StrictMath`. In virtually all cases this makes no difference in the return value, and the difference is very small, but may not exactly match `java.lang.Math` on x86 platforms in cases like, for example, `log(3.0)`, whose value varies between `Math.log()` and `StrictMath.log()`.
      Show
      The result of `java.lang.Math`'s `log`, `log1p`, `exp`, `expm1`, and `pow` may vary across platforms. In Spark 3.0, the result of the equivalent SQL functions (including related SQL functions like `LOG10`) return values consistent with `java.lang.StrictMath`. In virtually all cases this makes no difference in the return value, and the difference is very small, but may not exactly match `java.lang.Math` on x86 platforms in cases like, for example, `log(3.0)`, whose value varies between `Math.log()` and `StrictMath.log()`.

      Description

      Sorry to disturb again, we ran unit tests on arm64 instance, and there are other sql tests failed:

       - pgSQL/float8.sql *** FAILED ***
       Expected "{color:#f691b2}0.549306144334054[9]{color}", but got "{color:#f691b2}0.549306144334054[8]{color}" Result did not match for query #56
       SELECT atanh(double('0.5')) (SQLQueryTestSuite.scala:362)
       - pgSQL/numeric.sql *** FAILED ***
       Expected "2 {color:#59afe1}2247902679199174[72{color} 224790267919917955.1326161858
       4 7405685069595001 7405685069594999.0773399947
       5 5068226527.321263 5068226527.3212726541
       6 281839893606.99365 281839893606.9937234336
       7 {color:#d04437}1716699575118595840{color} 1716699575118597095.4233081991
       8 167361463828.0749 167361463828.0749132007
       9 {color:#14892c}107511333880051856]{color} 107511333880052007....", but got "2 {color:#59afe1}2247902679199174[40{color} 224790267919917955.1326161858
       4 7405685069595001 7405685069594999.0773399947
       5 5068226527.321263 5068226527.3212726541
       6 281839893606.99365 281839893606.9937234336
       7 {color:#d04437}1716699575118595580{color} 1716699575118597095.4233081991
       8 167361463828.0749 167361463828.0749132007
       9 {color:#14892c}107511333880051872]{color} 107511333880052007...." Result did not match for query #496
       SELECT t1.id1, t1.result, t2.expected
       FROM num_result t1, num_exp_power_10_ln t2
       WHERE t1.id1 = t2.id
       AND t1.result != t2.expected (SQLQueryTestSuite.scala:362)
      

      The first test failed, because the value of math.log(3.0) is different on aarch64:

      # on x86_64:

      scala> math.log(3.0)
      res50: Double = 1.0986122886681098
      

      # on aarch64:

      scala> math.log(3.0)
      res19: Double = 1.0986122886681096
      

      And I tried math.log(4.0), math.log(5.0) and they are same, I don't know why math.log(3.0) is so special? But the result is different indeed on aarch64.

      The second test failed, because some values of pow() is different on aarch64, according to the test, I took tests on aarch64 and x86_64, take '-83028485' as example:

      # on x86_64:

      scala> import java.lang.Math._
      import java.lang.Math._
      
      scala> abs(-83028485)
      res3: Int = 83028485
      
      scala> var a = -83028485
      a: Int = -83028485
      
      scala> abs(a)
      res4: Int = 83028485
      
      scala> math.log(abs(a))
      res5: Double = 18.234694299654787
      
      scala> pow(10, math.log(abs(a)))
      res6: Double ={color:#d04437} 1.71669957511859584E18{color}
      

      # on aarch64:

      scala> var a = -83028485
      a: Int = -83028485
      
      scala> abs(a)
      res38: Int = 83028485
      
      scala> math.log(abs(a))
      res39: Double = 18.234694299654787
      
      scala> pow(10, math.log(abs(a)))
      res40: Double = 1.71669957511859558E18
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srowen Sean R. Owen
                Reporter:
                huangtianhua huangtianhua
                Shepherd:
                Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: