[SPARK-19072] Catalyst's IN always returns false for infinity

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: SQL, Tests
• Assignee: Wenchen Fan
• Reporter: Kay Ousterhout
• Created: 04/Jan/17 00:48
• Updated: 04/Jan/17 06:40
• Resolved: 04/Jan/17 06:40

Description

This bug was caused by the fix for SPARK-18999 (https://github.com/apache/spark/pull/16402)

This can be reproduced by adding the following test to PredicateSuite.scala (which will consistently fail):

val value = NonFoldableLiteral(Double.PositiveInfinity, DoubleType)
checkEvaluation(In(value, List(value)), true)

This bug is causing org.apache.spark.sql.catalyst.expressions.PredicateSuite.IN to fail approximately 10% of the time (it fails anytime the value is Infinity or -Infinity and the correct answer is True – e.g., https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70826/testReport/org.apache.spark.sql.catalyst.expressions/PredicateSuite/IN/, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70830/console).

Issue Links

• links to: [Github] Pull Request #16469 (cloud-fan)