[SPARK-26950] Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.4, 2.4.2, 3.0.0
• Fix Version/s: 2.3.4, 2.4.1, 3.0.0
• Component/s: SQL, Tests
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 21/Feb/19 07:33
• Updated: 22/Feb/19 21:55
• Resolved: 22/Feb/19 04:28

Description

Apache Spark uses the predefined `Float.NaN` and `Double.NaN` for NaN values, but there exists more NaN values with different binary presentations.

scala> java.nio.ByteBuffer.allocate(4).putFloat(Float.NaN).array
res1: Array[Byte] = Array(127, -64, 0, 0)

scala> val x = java.lang.Float.intBitsToFloat(-6966608)
x: Float = NaN

scala> java.nio.ByteBuffer.allocate(4).putFloat(x).array
res2: Array[Byte] = Array(-1, -107, -78, -80)

`RandomDataGenerator` generates these NaN values. It's good, but it causes `checkEvaluationWithUnsafeProjection` failures due to the difference between `UnsafeRow` binary presentation. The following is the UT failure instance. This issue aims to fix this flakiness.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/102528/testReport/

Failed
org.apache.spark.sql.avro.AvroCatalystDataConversionSuite.flat schema struct<col_0:decimal(16,11),col_1:float,col_2:decimal(38,0),col_3:decimal(38,0),col_4:string> with seed -81044812370056695

Issue Links

• links to: GitHub Pull Request #23851