[SPARK-48928] Log Warning for Calling .unpersist() on Locally Checkpointed RDDs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Mingkang Li
• Reporter: Mingkang Li
• Created: 17/Jul/24 17:38
• Updated: 23/Jul/24 19:43
• Resolved: 23/Jul/24 19:42

Description

Summary:
This change proposes to log a warning message when the .unpersist() method is called on RDDs that have been locally checkpointed in Apache Spark. This aims to inform users about the potential risks of unpersisting such RDDs without altering the existing behavior of the method.

Background:
Local checkpointing in Spark truncates the lineage of an RDD, meaning that the RDD cannot be recomputed from its source. If an RDD that has been locally checkpointed is unpersisted, it loses its data and cannot be regenerated. This can lead to job failures if subsequent actions or transformations are attempted on the unpersisted RDD.

Proposed Change:
To mitigate this issue, a warning message will be logged whenever .unpersist() is called on a locally checkpointed RDD. This approach maintains the current functionality while alerting users to the potential consequences of their actions. This change is intended to be non-disruptive and is a step towards better user awareness and debugging.

Issue Links

• links to: GitHub Pull Request #47391