[SPARK-29782] spark broadcast can not be destoryed in some versions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.3, 2.4.1, 2.4.2, 2.4.3, 2.4.4
• Component/s: Block Manager, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: tzxxh
• Created: 07/Nov/19 01:33
• Updated: 25/May/21 01:55
• Resolved: 25/May/21 01:43

Description

In spark version (2.3.3 , 2.4.1 , 2.4.2 , 2.4.3 , 2.4.4) use Broadcast.destroy() method can not destroy the broadcast data, the driver and executor storage memory in spark ui is continuous increase。

val batch = Seq(1 to 9999: _*) 
val strSeq = batch.map(i => s"xxh-$i") 
val rdd = sc.parallelize(strSeq) 
rdd.cache() 
batch.foreach(_ => { 
  val broc = sc.broadcast(strSeq) 
  rdd.map(id => broc.value.contains(id)).collect() 
  broc.destroy() 
})