[SPARK-28519] Tests failed on aarch64 due the value of math.log and power function is different

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Labels: release-notes

Docs Text

The result of `java.lang.Math`'s `log`, `log1p`, `exp`, `expm1`, and `pow` may vary across platforms. In Spark 3.0, the result of the equivalent SQL functions (including related SQL functions like `LOG10`) return values consistent with `java.lang.StrictMath`. In virtually all cases this makes no difference in the return value, and the difference is very small, but may not exactly match `java.lang.Math` on x86 platforms in cases like, for example, `log(3.0)`, whose value varies between `Math.log()` and `StrictMath.log()`.

• Assignee: Sean R. Owen
• Reporter: huangtianhua
• Shepherd: Dongjoon Hyun
• Created: 26/Jul/19 07:00
• Updated: 12/Dec/22 18:10
• Resolved: 02/Aug/19 15:55

Description

Sorry to disturb again, we ran unit tests on arm64 instance, and there are other sql tests failed:

 - pgSQL/float8.sql *** FAILED ***
 Expected "{color:#f691b2}0.549306144334054[9]{color}", but got "{color:#f691b2}0.549306144334054[8]{color}" Result did not match for query #56
 SELECT atanh(double('0.5')) (SQLQueryTestSuite.scala:362)
 - pgSQL/numeric.sql *** FAILED ***
 Expected "2 {color:#59afe1}2247902679199174[72{color} 224790267919917955.1326161858
 4 7405685069595001 7405685069594999.0773399947
 5 5068226527.321263 5068226527.3212726541
 6 281839893606.99365 281839893606.9937234336
 7 {color:#d04437}1716699575118595840{color} 1716699575118597095.4233081991
 8 167361463828.0749 167361463828.0749132007
 9 {color:#14892c}107511333880051856]{color} 107511333880052007....", but got "2 {color:#59afe1}2247902679199174[40{color} 224790267919917955.1326161858
 4 7405685069595001 7405685069594999.0773399947
 5 5068226527.321263 5068226527.3212726541
 6 281839893606.99365 281839893606.9937234336
 7 {color:#d04437}1716699575118595580{color} 1716699575118597095.4233081991
 8 167361463828.0749 167361463828.0749132007
 9 {color:#14892c}107511333880051872]{color} 107511333880052007...." Result did not match for query #496
 SELECT t1.id1, t1.result, t2.expected
 FROM num_result t1, num_exp_power_10_ln t2
 WHERE t1.id1 = t2.id
 AND t1.result != t2.expected (SQLQueryTestSuite.scala:362)

The first test failed, because the value of math.log(3.0) is different on aarch64:

# on x86_64:

scala> math.log(3.0)
res50: Double = 1.0986122886681098

# on aarch64:

scala> math.log(3.0)
res19: Double = 1.0986122886681096

And I tried math.log(4.0), math.log(5.0) and they are same, I don't know why math.log(3.0) is so special? But the result is different indeed on aarch64.

The second test failed, because some values of pow() is different on aarch64, according to the test, I took tests on aarch64 and x86_64, take '-83028485' as example:

# on x86_64:

scala> import java.lang.Math._
import java.lang.Math._

scala> abs(-83028485)
res3: Int = 83028485

scala> var a = -83028485
a: Int = -83028485

scala> abs(a)
res4: Int = 83028485

scala> math.log(abs(a))
res5: Double = 18.234694299654787

scala> pow(10, math.log(abs(a)))
res6: Double ={color:#d04437} 1.71669957511859584E18{color}

# on aarch64:

scala> var a = -83028485
a: Int = -83028485

scala> abs(a)
res38: Int = 83028485

scala> math.log(abs(a))
res39: Double = 18.234694299654787

scala> pow(10, math.log(abs(a)))
res40: Double = 1.71669957511859558E18

Issue Links

• is part of: SPARK-29106 Add jenkins arm test for spark
• links to: GitHub Pull Request #25279