[SPARK-11737] String may not be serialized correctly with Kyro

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.0
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: SQL
• Target Version/s: 1.5.3, 1.6.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 13/Nov/15 20:47
• Updated: 11/Apr/16 08:08
• Resolved: 18/Nov/15 03:50

Description

When run in cluster mode, the driver may have different memory (and configs) than executor, also if Kyro is used, then string can not be collected back to driver:

>>> sqlContext.range(10).selectExpr("repeat(cast(id as string), 9)").show()
+----------------------------+
|repeat(cast(id as string),9)|
+----------------------------+
|                   	0|
|                   	1|
|                   	2|
|                   	3|
|                   	4|
|                   	5|
|                   	6|
|                   	7|
|                   	8|
|                   	9|
+----------------------------+

Issue Links

• duplicates: SPARK-11657 Bad Dataframe data read from parquet
• duplicates: SPARK-14524 In SparkSQL, it can't be select column of String type because of UTF8String when setting more than 32G for executors.
• links to: [Github] Pull Request #9704 (davies)