[SPARK-12414] Remove closure serializer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Sean R. Owen
• Reporter: Andrew Or
• Created: 17/Dec/15 23:37
• Updated: 19/Jun/17 18:29
• Resolved: 10/Feb/16 21:34

Description

There is a config `spark.closure.serializer` that accepts exactly one value: the java serializer. This is because there are currently bugs in the Kryo serializer that make it not a viable candidate. This was uncovered by an unsuccessful attempt to make it work: SPARK-7708.

My high level point is that the Java serializer has worked well for at least 6 Spark versions now, and it is an incredibly complicated task to get other serializers (not just Kryo) to work with Spark's closures. IMO the effort is not worth it and we should just remove this documentation and all the code associated with it.

Issue Links

• supercedes: SPARK-12415 Do not use closure serializer to serialize task result
• links to: [Github] Pull Request #11150 (srowen)