[SPARK-2897] org.apache.spark.broadcast.TorrentBroadcast does use the serializer class specified in the spark option "spark.serializer"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.2, 1.1.0
• Fix Version/s: 1.0.3, 1.1.0
• Component/s: Spark Core
• Target Version/s: 1.0.3, 1.1.0
• Assignee: Guoqiang Li
• Reporter: Joseph Su
• Created: 07/Aug/14 04:51
• Updated: 08/Aug/14 23:59
• Resolved: 08/Aug/14 23:59

Description

HTTPBroadcast will changes the serializer according to the setting in "spark.serializer".

However, TorrentBroadcast does not read that option. It always uses Java built-in serializer. ( Specifically, it uses org.apache.spark.util.Utils.serialize() and org.apache.spark.util.Utils.deserialize() ).

With TorrentBroadcast, Spark programs can not broadcast objects that can be serialized with custom Kryo serializer but not with Java built-in serializer.

Issue Links

• is duplicated by: SPARK-2928 TorrentBroadcast should use the user specified serializer
• is related to: SPARK-2920 TorrentBroadcast does not support broadcast compression
• links to: [Github] Pull Request #1836 (witgo)