[SPARK-6368] Build a specialized serializer for Exchange operator.

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SQL
• Target Version/s: 1.4.0
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 16/Mar/15 23:42
• Updated: 01/May/15 22:16
• Resolved: 21/Apr/15 01:43

Description

Kryo is still pretty slow because it works on individual objects and relative expensive to allocate. For Exchange operator, because the schema for key and value are already defined, we can create a specialized serializer to handle the specific schemas of key and value.

Issue Links

• is blocked by: SPARK-6367 Use the proper data type for those expressions that are hijacking existing data types.
• is related to: SPARK-6986 Makes SparkSqlSerializer2 support sort-based shuffle with sort merge
• links to: [Github] Pull Request #5497 (yhuai)
• links to: [Github] Pull Request #5849 (yhuai)