[SPARK-50364] Row.jsonValue is not able to serialize LocalDateTime

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.3
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Karim Ramadan
• Reporter: Karim Ramadan
• Created: 20/Nov/24 10:51
• Updated: 20/Nov/24 16:23
• Resolved: 20/Nov/24 16:01

Description

Trying to serialize to JSON a Row containing a column of type TimestampNTZType (LocalDateTime) with methods:

def json: String = compact(jsonValue)
def prettyJson: String = pretty(render(jsonValue)) 

Will fail with:

[FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G
org.apache.spark.SparkIllegalArgumentException: [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G
	at org.apache.spark.sql.Row.toJson$1(Row.scala:663)
	at org.apache.spark.sql.Row.toJson$1(Row.scala:651)
	at org.apache.spark.sql.Row.jsonValue(Row.scala:665)
	at org.apache.spark.sql.Row.jsonValue$(Row.scala:598)
	at org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:28)
	at org.apache.spark.sql.RowJsonSuite.$anonfun$testJson$1(RowJsonSuite.scala:41)

This happens because the toJson method inside of the class is missing a serializer for the LocalDateTime type

    def toJson(value: Any, dataType: DataType): JValue = (value, dataType) match {
      case (null, _) => JNull
      case (b: Boolean, _) => JBool(b)
      case (b: Byte, _) => JLong(b)
      case (s: Short, _) => JLong(s)
      case (i: Int, _) => JLong(i)
      case (l: Long, _) => JLong(l)
...

Issue Links

• is duplicated by: SPARK-50365 Row.jsonValue is not able to serialize LocalDateTime
• links to: GitHub Pull Request #48905