[SPARK-38990] date_trunc and trunc both fail with format from column in inline table

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.2.1, 3.3.0, 3.4.0
• Fix Version/s: 3.1.3, 3.0.4, 3.3.0, 3.2.2
• Component/s: SQL
• Assignee: Bruce Robbins
• Reporter: Bruce Robbins
• Created: 21/Apr/22 22:27
• Updated: 12/Dec/22 18:10
• Resolved: 22/Apr/22 03:42

Description

This fails:

create or replace temp view v1 as
select * from values ('week', timestamp'2012-01-01') as data(col1, col2);

select date_trunc(col1, col2) from v1;

It fails with a NullPointerException:

java.lang.NullPointerException: null
	at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getAccessor$8(InternalRow.scala:141) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getAccessor$8$adapted(InternalRow.scala:141) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.BoundReference.eval(BoundAttribute.scala:40) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.TruncInstant.evalHelper(datetimeExpressions.scala:2117) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.TruncInstant.evalHelper$(datetimeExpressions.scala:2112) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.TruncTimestamp.evalHelper(datetimeExpressions.scala:2277) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.TruncTimestamp.eval(datetimeExpressions.scala:2295) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:157) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(InterpretedMutableProjection.scala:97) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$47.$anonfun$applyOrElse$77(Optimizer.scala:1987) ~[spark-catalyst_2.12-3

 However, if you cache v1, then it works:

spark-sql> cache table v1;
Time taken: 2.086 seconds
spark-sql> select date_trunc(col1, col2) from v1;
2011-12-26 00:00:00
Time taken: 0.172 seconds, Fetched 1 row(s)
spark-sql> 

Both date_trunc and trunc exhibit this behavior.

Issue Links

• links to: [Github] Pull Request #36312 (bersprockets)