[SPARK-17930] The SerializerInstance instance used when deserializing a TaskResult is not reused

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1, 2.0.1
• Fix Version/s: 2.1.0
• Component/s: Spark Core
• Assignee: Guoqiang Li
• Reporter: Guoqiang Li
• Created: 14/Oct/16 03:25
• Updated: 18/Oct/16 20:47
• Resolved: 18/Oct/16 20:47

Description

The following code is called when the DirectTaskResult instance is deserialized

  def value(): T = {
    if (valueObjectDeserialized) {
      valueObject
    } else {
      // Each deserialization creates a new instance of SerializerInstance, which is very time-consuming
      val resultSer = SparkEnv.get.serializer.newInstance()
      valueObject = resultSer.deserialize(valueBytes)
      valueObjectDeserialized = true
      valueObject
    }
  }

Issue Links

• links to: [Github] Pull Request #15512 (witgo)