[SPARK-50063] Support Variant in Spark Connect Scala client

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect
• Assignee: Unassigned
• Reporter: Takuya Ueshin
• Created: 2 days ago 23:02
• Updated: 2 days ago 23:02

Description

Currently Spark Connect Scala client doesn't support Variant type when collecting data.

scala> spark.sql("select parse_json('1')").collect()
org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_DATATYPE] Unsupported data type "VARIANT". SQLSTATE: 0A000
  org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError(ExecutionErrors.scala:159)
  org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError$(ExecutionErrors.scala:156)
  org.apache.spark.sql.errors.ExecutionErrors$.unsupportedDataTypeError(ExecutionErrors.scala:254)
  org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:370)
...