[SPARK-19322] Allow customizable column name in ScalaReflection.schemaFor

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Wei Liu
• Created: 21/Jan/17 17:02
• Updated: 21/May/19 04:14
• Resolved: 21/May/19 04:14

Description

When creating StructType using ScalaRelfection.schemaFor, the column names are defined by scala case class parameter names. This causes minor inconvenience when reading/writing data between Python and Scala which follow different naming conventions. Ideally, we should allow customize column names using Scala annotations in a similar fashion to Jackson serialization libraries.

For example, we could define schema as :

case class MySchema(@fieldName("long_field") longField: Long, @fieldName("int_field") intField: Int)

The field/column names can now be customized.