[SPARK-29810] Missing persist on retaggedInput in RandomForest.run()

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.3
• Component/s: ML
• Assignee: Unassigned
• Reporter: IcySanwitch
• Created: 09/Nov/19 09:36
• Updated: 16/Nov/19 20:42
• Resolved: 10/Nov/19 19:20

Description

The rdd retaggedInput should be persisted in ml.tree.impl.RandomForest.run(), because it will be used more than one actions.

  def run(
      input: RDD[LabeledPoint],
      strategy: OldStrategy,
      numTrees: Int,
      featureSubsetStrategy: String,
      seed: Long,
      instr: Option[Instrumentation],
      prune: Boolean = true, // exposed for testing only, real trees are always pruned
      parentUID: Option[String] = None): Array[DecisionTreeModel] = {

    val timer = new TimeTracker()
    timer.start("total")
    timer.start("init")
    val retaggedInput = input.retag(classOf[LabeledPoint]) // it needs to be persisted

This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses.

Issue Links

• duplicates: SPARK-29818 Missing persist on RDD