[SPARK-28686] Higher precision in Math.toRadians from JDK9+

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 11/Aug/19 13:30
• Updated: 13/Aug/19 03:11
• Resolved: 12/Aug/19 09:25

Description

- udf_radians *** FAILED *** (1 second, 791 milliseconds) from HiveCompatibilitySuite.

[info] - udf_radians *** FAILED *** (1 second, 791 milliseconds)
[info]   Results do not match for udf_radians:
[info]   == Parsed Logical Plan ==
[info]   'Project [unresolvedalias('radians(57.2958), None)]
[info]   +- 'GlobalLimit 1
[info]      +- 'LocalLimit 1
[info]         +- 'UnresolvedRelation [src]
[info]
[info]   == Analyzed Logical Plan ==
[info]   RADIANS(CAST(57.2958 AS DOUBLE)): double
[info]   Project [RADIANS(cast(57.2958 as double)) AS RADIANS(CAST(57.2958 AS DOUBLE))#93209]
[info]   +- GlobalLimit 1
[info]      +- LocalLimit 1
[info]         +- SubqueryAlias `default`.`src`
[info]            +- HiveTableRelation `default`.`src`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [key#93207, value#93208]
[info]
[info]   == Optimized Logical Plan ==
[info]   Project [1.0000003575641672 AS RADIANS(CAST(57.2958 AS DOUBLE))#93209]
[info]   +- GlobalLimit 1
[info]      +- LocalLimit 1
[info]         +- Project
[info]            +- InMemoryRelation [key#93207, value#93208], StorageLevel(disk, memory, deserialized, 1 replicas)
[info]                  +- Scan hive default.src [key#93145, value#93146], HiveTableRelation `default`.`src`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [key#93145, value#93146]
[info]
[info]   == Physical Plan ==
[info]   *(2) Project [1.0000003575641672 AS RADIANS(CAST(57.2958 AS DOUBLE))#93209]
[info]   +- *(2) GlobalLimit 1
[info]      +- Exchange SinglePartition, true
[info]         +- *(1) LocalLimit 1
[info]            +- Scan In-memory table src
[info]                  +- InMemoryRelation [key#93207, value#93208], StorageLevel(disk, memory, deserialized, 1 replicas)
[info]                        +- Scan hive default.src [key#93145, value#93146], HiveTableRelation `default`.`src`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [key#93145, value#93146]
[info]
[info]   RADIANS(CAST(57.2958 AS DOUBLE))
[info]   !== HIVE - 1 row(s) ==   == CATALYST - 1 row(s) ==
[info]   !1.000000357564167       1.0000003575641672 (HiveComparisonTest.scala:437)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
[info]   at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info]   at org.scalatest.Assertions.fail(Assertions.scala:1089)
[info]   at org.scalatest.Assertions.fail$(Assertions.scala:1085)
[info]   at org.scalatest.FunSuite.fail(FunSuite.scala:1560)
[info]   at org.apache.spark.sql.hive.execution.HiveComparisonTest.$anonfun$createQueryTest$33(HiveComparisonTest.scala:437)
[info]   at org.apache.spark.sql.hive.execution.HiveComparisonTest.$anonfun$createQueryTest$33$adapted(HiveComparisonTest.scala:369)
[info]   at scala.runtime.Tuple3Zipped$.$anonfun$foreach$1(Tuple3Zipped.scala:124)
[info]   at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
[info]   at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
[info]   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:39)
[info]   at scala.runtime.Tuple3Zipped$.foreach$extension(Tuple3Zipped.scala:122)
[info]   at org.apache.spark.sql.hive.execution.HiveComparisonTest.doTest$1(HiveComparisonTest.scala:369)
[info]   at org.apache.spark.sql.hive.execution.HiveComparisonTest.$anonfun$createQueryTest$10(HiveComparisonTest.scala:470)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
[info]   at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
[info]   at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info]   at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
[info]   at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
[info]   at org.apache.spark.sql.hive.execution.HiveCompatibilitySuite.org$scalatest$BeforeAndAfter$$super$runTest(HiveCompatibilitySuite.scala:33)
[info]   at org.scalatest.BeforeAndAfter.runTest(BeforeAndAfter.scala:203)
[info]   at org.scalatest.BeforeAndAfter.runTest$(BeforeAndAfter.scala:192)
[info]   at org.apache.spark.sql.hive.execution.HiveCompatibilitySuite.runTest(HiveCompatibilitySuite.scala:33)
[info]   at org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
[info]   at scala.collection.immutable.List.foreach(List.scala:392)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info]   at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
[info]   at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
[info]   at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
[info]   at org.scalatest.Suite.run(Suite.scala:1147)
[info]   at org.scalatest.Suite.run$(Suite.scala:1129)
[info]   at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
[info]   at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info]   at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
[info]   at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.sql.hive.execution.HiveCompatibilitySuite.org$scalatest$BeforeAndAfter$$super$run(HiveCompatibilitySuite.scala:33)
[info]   at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:258)
[info]   at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:256)
[info]   at org.apache.spark.sql.hive.execution.HiveCompatibilitySuite.run(HiveCompatibilitySuite.scala:33)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:296)
[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[info]   at java.base/java.lang.Thread.run(Thread.java:834)

How to reproduce:

public class TestRadians {
  public static void main(String[] args) {
    System.out.println(java.lang.Math.toRadians(57.2958));
  }
}

[root@spark-3267648 ~]# javac TestRadians.java
[root@spark-3267648 ~]# /usr/lib/jdk-11.0.3/bin/java TestRadians
1.0000003575641672
[root@spark-3267648 ~]# /usr/lib/jdk8u222-b10/bin/java TestRadians
1.000000357564167

Issue Links

• links to: GitHub Pull Request #25417