[SPARK-25178] Directly ship the StructType objects of the keySchema / valueSchema for xxxHashMapGenerator

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Kazuaki Ishizaki
• Reporter: Kris Mok
• Created: 21/Aug/18 18:09
• Updated: 24/Aug/18 06:00
• Resolved: 24/Aug/18 06:00

Description

Following SPARK-18952 and SPARK-22273, this ticket proposes to change the generated field name of the keySchema / valueSchema to a dummy name instead of using key.name.

In previous discussion from SPARK-18952's PR [1], it was already suggested that the field names were being used, so it's not worth capturing the strings as reference objects here. Josh suggested merging the original fix as-is due to backportability / pickability concerns. Now that we're coming up to a new release, this can be revisited.

[1]: https://github.com/apache/spark/pull/16361#issuecomment-270253719

Issue Links

• links to: [Github] Pull Request #22187 (kiszk)