[SPARK-36854] Parquet reader fails on load of ANSI interval when off-heap is enabled

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 25/Sep/21 19:09
• Updated: 12/Dec/22 18:10
• Resolved: 26/Sep/21 03:46

Description

When off-heap column vector is enabled, parquet reader fails. The example below portraits the issue:

scala> Seq(java.time.Period.of)
scala> val df = Seq(java.time.Period.ofMonths(1)).toDF
df: org.apache.spark.sql.DataFrame = [value: interval year to month]

scala> df.write.parquet("/Users/maximgekk/tmp/parquet_offheap")
scala> spark.conf.set("spark.sql.columnVector.offheap.enabled", true)

scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap")
res2: org.apache.spark.sql.DataFrame = [value: interval year to month]

scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap").show()
21/09/25 22:09:03 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.RuntimeException: Unhandled YearMonthIntervalType(0,1)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.reserveInternal(OffHeapColumnVector.java:562)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.<init>(OffHeapColumnVector.java:75)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:53)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:42)

 

Issue Links

• links to: [Github] Pull Request #34106 (MaxGekk)
• links to: [Github] Pull Request #34106 (MaxGekk)