[SPARK-29824] Missing persist on trainDataset in ml.classification.GBTClassifier.train()

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.3
• Component/s: ML
• Assignee: Unassigned
• Reporter: IcySanwitch
• Created: 10/Nov/19 12:28
• Updated: 10/Nov/19 19:20
• Resolved: 10/Nov/19 19:20

Description

The rdd trainDataset in ml.classification.GBTClassifier.train() is used by an action first and other actions in GradientBoostedTrees.run/runWithValidation, but it is not persisted, which will cause recomputation on trainDataset.

  override protected def train(
      dataset: Dataset[_]): GBTClassificationModel = instrumented { instr =>
    val categoricalFeatures: Map[Int, Int] =
      MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol)))
    ...
    val numFeatures = trainDataset.first().features.size // first use trainDataset
    ...
    // trainDataset will be used by other actions in run methods.    
    val (baseLearners, learnerWeights) = if (withValidation) {
      GradientBoostedTrees.runWithValidation(trainDataset, validationDataset, boostingStrategy,
        $(seed), $(featureSubsetStrategy))
    } else {
      GradientBoostedTrees.run(trainDataset, boostingStrategy, $(seed), $(featureSubsetStrategy))
    }

This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses.

Issue Links

• duplicates: SPARK-29818 Missing persist on RDD