[SPARK-9139] Add backwards-compatibility tests for DataType.fromJson()

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 17/Jul/15 19:43
• Updated: 21/May/19 04:13
• Resolved: 21/May/19 04:13

Description

SQL's DataType.fromJson is a public API and thus must be backwards-compatible; there are also backwards-compatibility concerns related to persistence of DataType JSON in metastores.

Unfortunately, we do not have any backwards-compatibility tests which attempt to read old JSON values that were written by earlier versions of Spark. DataTypeSuite has "roundtrip" tests that test fromJson(toJson(foo)), but this doesn't ensure compatibility.

I think that we should address this by capuring the JSON strings produced in Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes from those strings.

This might be a good starter task for someone who wants to contribute to SQL tests.