[SPARK-29828] Missing persist on ratings in ml.recommendation.ALS.train

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.3
• Component/s: ML
• Assignee: Unassigned
• Reporter: IcySanwitch
• Created: 10/Nov/19 14:43
• Updated: 10/Nov/19 19:21
• Resolved: 10/Nov/19 19:21

Description

There is a ratings.isEmpty() at the beginning of theml.recommendation.ALS.train(). Actually, isEmpty() has an action operator.

  def isEmpty(): Boolean = withScope {
    partitions.length == 0 || take(1).length == 0
  }

So rdd ratings will be used by multi actions, it should be persisted, and unpersisted after its child rdd has been persisted.

  def train[ID: ClassTag]( // scalastyle:ignore
      ratings: RDD[Rating[ID]],
    ...
    require(!ratings.isEmpty(), s"No ratings available from $ratings") // first use ratings
    ...
    val blockRatings = partitionRatings(ratings, userPart, itemPart)
      .persist(intermediateRDDStorageLevel)
    val (userInBlocks, userOutBlocks) =
      makeBlocks("user", blockRatings, userPart, itemPart, intermediateRDDStorageLevel)
    userOutBlocks.count()    // materialize blockRatings and user blocks
    // ratings should be unpersisted here

This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses.

Issue Links

• duplicates: SPARK-29818 Missing persist on RDD