Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29782

spark broadcast can not be destoryed in some versions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.3, 2.4.1, 2.4.2, 2.4.3, 2.4.4
    • Fix Version/s: None
    • Component/s: Block Manager, Spark Core
    • Labels:
      None

      Description

      In spark version (2.3.3 , 2.4.1 , 2.4.2 , 2.4.3 , 2.4.4) use Broadcast.destroy() method can not destroy the broadcast data, the driver and executor storage memory in spark ui is continuous increase。

      val batch = Seq(1 to 9999: _*) 
      val strSeq = batch.map(i => s"xxh-$i") 
      val rdd = sc.parallelize(strSeq) 
      rdd.cache() 
      batch.foreach(_ => { 
        val broc = sc.broadcast(strSeq) 
        rdd.map(id => broc.value.contains(id)).collect() 
        broc.destroy() 
      })
      

       

        Attachments

        1. problem versions.png
          8 kB
          tzxxh
        2. correct version.png
          5 kB
          tzxxh

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tzxxh tzxxh
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: