[SPARK-22442] Schema generated by Product Encoder doesn't match case class field name when using non-standard characters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.2, 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• Target Version/s: 2.2.1, 2.3.0
• Assignee: L. C. Hsieh
• Reporter: Mikel San Vicente
• Created: 03/Nov/17 20:54
• Updated: 13/Nov/17 06:32
• Resolved: 09/Nov/17 10:55

Description

Product encoder encodes special characters wrongly when field name contains certain nonstandard characters.

For example for:

case class MyType(`field.1`: String, `field 2`: String)

we will get the following schema

root

– field$u002E1: string (nullable = true)

– field$u00202: string (nullable = true)

As a consequence of this issue a DataFrame with the correct schema can't be converted to a Dataset using .as[MyType]

Issue Links

• links to: [Github] Pull Request #19664 (viirya)
• links to: [Github] Pull Request #19734 (viirya)
• links to: [Github] Pull Request #19736 (viirya)