[SPARK-39983] Should not cache unserialized broadcast relations on the driver

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Alex Balikov
• Reporter: Alex Balikov
• Created: 04/Aug/22 20:55
• Updated: 28/Mar/24 00:55
• Resolved: 11/Aug/22 00:14

Description

In TorrentBroadcast.writeBlocks we store the unserialized broadcast object in addition to the serialized version of it - 

private def writeBlocks(value: T): Int = {
    import StorageLevel._
    // Store a copy of the broadcast variable in the driver so that tasks run on the driver
    // do not create a duplicate copy of the broadcast variable's value.
    val blockManager = SparkEnv.get.blockManager
    if (!blockManager.putSingle(broadcastId, value, MEMORY_AND_DISK, tellMaster = false)) {
      throw new SparkException(s"Failed to store $broadcastId in BlockManager")
    }
 

In case of broadcast relations, these objects can be fairly large (60MB in one observed case) and are not strictly necessary on the driver.

Add the option to not keep the unserialized versions of the objects.

 

Issue Links

• links to: [Github] Pull Request #37413 (alex-balikov)
• links to: [Github] Pull Request #37413 (alex-balikov)
• links to: GitHub Pull Request #5149