[SPARK-33945] Handles a random seed consisting of an expr tree

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.8, 3.0.2, 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Takeshi Yamamuro
• Reporter: Takeshi Yamamuro
• Created: 30/Dec/20 12:24
• Updated: 04/Jan/21 05:37
• Resolved: 04/Jan/21 05:37

Description

This ticket aims at fixing the minor bug that throws an analysis exception when a seed param in `rand`/`randn` having a expr tree (e.g., `rand(1 + 1)`) with constant folding (`ConstantFolding` and `ReorderAssociativeOperator`) disabled. A query to reproduce this issue is as follows;

// v3.1.0, v3.0.2, and v2.4.8
$./bin/spark-shell 
scala> sql("select rand(1 + 2)").show()
+-------------------+
|      rand((1 + 2))|
+-------------------+
|0.25738143505962285|
+-------------------+

$./bin/spark-shell --conf spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding,org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator
scala> sql("select rand(1 + 2)").show()
org.apache.spark.sql.AnalysisException: Input argument to rand must be an integer, long or null literal.;
  at org.apache.spark.sql.catalyst.expressions.RDG.seed$lzycompute(randomExpressions.scala:49)
  at org.apache.spark.sql.catalyst.expressions.RDG.seed(randomExpressions.scala:46)
  at org.apache.spark.sql.catalyst.expressions.Rand.doGenCode(randomExpressions.scala:98)
  at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:146)
  at scala.Option.getOrElse(Option.scala:189)
  ...

A root cause is that the match-case code below cannot handle the case described above:
https://github.com/apache/spark/blob/42f5e62403469cec6da680b9fbedd0aa508dcbe5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala#L46-L51

Issue Links

• links to: [Github] Pull Request #30977 (maropu)