[SPARK-9332] CatalystTypeConverters.toScala does not work on UnsafeRows

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: Reynold Xin
• Reporter: Josh Rosen
• Created: 25/Jul/15 01:06
• Updated: 27/Jul/15 20:41
• Resolved: 27/Jul/15 20:41

Description

When CatalystTypeConverters.toScala is applied to an UnsafeRow it may lead to an UnsupportedOperationException when it tries to use a generic getter to retrieve primitive columns.

For example:

== Parsed Logical Plan ==
Sort [c183#208 ASC,c183#208 ASC], true
 Project [c183#208,c184#209]
  LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2

== Analyzed Logical Plan ==
c183: binary, c184: int
Sort [c183#208 ASC,c183#208 ASC], true
 Project [c183#208,c184#209]
  LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2

== Optimized Logical Plan ==
Sort [c183#208 ASC,c183#208 ASC], true
 LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2

== Physical Plan ==
UnsafeExternalSort [c183#208 ASC,c183#208 ASC], true, 0
 Exchange rangepartitioning(c183#208 ASC,c183#208 ASC)
  PhysicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2

java.lang.UnsupportedOperationException
	at org.apache.spark.sql.catalyst.expressions.UnsafeRow.get(UnsafeRow.java:223)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$IdentityConverter$.toScalaImpl(CatalystTypeConverters.scala:143)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:118)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:266)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:233)
	at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:386)
	at org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
	at org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)

Issue Links

• duplicates: SPARK-9368 Support get(ordinal, dataType) generic getter in UnsafeRow