[SPARK-41118] to_number/try_to_number throws NullPointerException when format is null

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.2, 3.4.0
• Component/s: SQL
• Assignee: Bruce Robbins
• Reporter: Bruce Robbins
• Created: 12/Nov/22 17:16
• Updated: 12/Dec/22 18:10
• Resolved: 17/Nov/22 00:56

Description

Example:

spark-sql> SELECT to_number('454', null);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
	at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:498)
	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
...
Caused by: java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat$lzycompute(numberFormatExpressions.scala:72)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat(numberFormatExpressions.scala:72)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter$lzycompute(numberFormatExpressions.scala:73)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter(numberFormatExpressions.scala:73)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.checkInputDataTypes(numberFormatExpressions.scala:81)

Also:

spark-sql> SELECT try_to_number('454', null);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
	at org.apache.spark.sql.execution.QueryExecution$.toInternalError(QueryExecution.scala:498)
	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
...
Caused by: java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat$lzycompute(numberFormatExpressions.scala:72)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormat(numberFormatExpressions.scala:72)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter$lzycompute(numberFormatExpressions.scala:73)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.numberFormatter(numberFormatExpressions.scala:73)
	at org.apache.spark.sql.catalyst.expressions.ToNumber.checkInputDataTypes(numberFormatExpressions.scala:81)
	at org.apache.spark.sql.catalyst.expressions.TryToNumber.checkInputDataTypes(numberFormatExpressions.scala:146)

Compare to to_binary and try_to_binary:

spark-sql> SELECT to_binary('abc', null);
NULL
Time taken: 3.111 seconds, Fetched 1 row(s)
spark-sql> SELECT try_to_binary('abc', null);
NULL
Time taken: 0.06 seconds, Fetched 1 row(s)
spark-sql>

Issue Links

• links to: [Github] Pull Request #38635 (bersprockets)
• links to: [Github] Pull Request #38697 (bersprockets)
• links to: [Github] Pull Request #38697 (bersprockets)