[SPARK-29756] CountVectorizer forget to unpersist intermediate rdd

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: ML
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 05/Nov/19 09:56
• Updated: 08/Nov/19 10:32
• Resolved: 08/Nov/19 10:32

Description

scala> val df = spark.createDataFrame(Seq(
     |       (0, Array("a", "b", "c")),
     |       (1, Array("a", "b", "b", "c", "a"))
     |     )).toDF("id", "words")
df: org.apache.spark.sql.DataFrame = [id: int, words: array<string>]scala> 
import org.apache.spark.ml.feature._
import org.apache.spark.ml.feature._

scala> val cvModel: CountVectorizerModel = new CountVectorizer().setInputCol("words").setOutputCol("features").setVocabSize(3).setMinDF(2).fit(df)
cvModel: org.apache.spark.ml.feature.CountVectorizerModel = cntVec_5edcfe4828c2

scala> sc.getPersistentRDDs
res0: scala.collection.Map[Int,org.apache.spark.rdd.RDD[_]] = Map(9 -> MapPartitionsRDD[9] at map at CountVectorizer.scala:223)
 

Issue Links

• links to: GitHub Pull Request #26398