[SPARK-17931] taskScheduler has some unneeded serialization

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Scheduler, Spark Core
• Assignee: Kay Ousterhout
• Reporter: Guoqiang Li
• Created: 14/Oct/16 03:56
• Updated: 17/May/20 17:48
• Resolved: 06/Jan/17 16:52

Description

In the existing code, there are three layers of serialization
involved in sending a task from the scheduler to an executor:

• A Task object is serialized
• The Task object is copied to a byte buffer that also
  contains serialized information about any additional JARs,
  files, and Properties needed for the task to execute. This
  byte buffer is stored as the member variable serializedTask
  in the TaskDescription class.
• The TaskDescription is serialized (in addition to the serialized
  task + JARs, the TaskDescription class contains the task ID and
  other metadata) and sent in a LaunchTask message.

While it is necessary to have two layers of serialization, so that
the JAR, file, and Property info can be deserialized prior to
deserializing the Task object, the third layer of deserialization is
unnecessary (this is as a result of SPARK-2521). We should
eliminate a layer of serialization by moving the JARs, files, and Properties
into the TaskDescription class.

Issue Links

• is related to: SPARK-19796 taskScheduler fails serializing long statements received by thrift server
• relates to: SPARK-18890 Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)
• links to: [Github] Pull Request #15505 (witgo)
• links to: [Github] Pull Request #16053 (kayousterhout)