[SPARK-12744] Inconsistent behavior parsing JSON with unix timestamp values

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: SQL
• Labels: release_notes releasenotes
• Target Version/s: 2.0.0
• Assignee: Anatoliy Plastinin
• Reporter: Anatoliy Plastinin
• Created: 10/Jan/16 22:06
• Updated: 11/Jan/16 20:57
• Resolved: 11/Jan/16 18:30

Description

Let’s have following json

val rdd = sc.parallelize("""{"ts":1452386229}""" :: Nil)

Spark sql casts int to timestamp treating int value as a number of seconds.
https://issues.apache.org/jira/browse/SPARK-11724

scala> sqlContext.read.json(rdd).select($"ts".cast(TimestampType)).show
+--------------------+
|                  ts|
+--------------------+
|2016-01-10 01:37:...|
+--------------------+

However parsing json with schema gives different result

scala> val schema = (new StructType).add("ts", TimestampType)
schema: org.apache.spark.sql.types.StructType = StructType(StructField(ts,TimestampType,true))

scala> sqlContext.read.schema(schema).json(rdd).show
+--------------------+
|                  ts|
+--------------------+
|1970-01-17 20:26:...|
+--------------------+

Issue Links

• links to: [Github] Pull Request #10687 (antlypls)