[SPARK-45386] Correctness issue when persisting using StorageLevel.NONE

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 3.5.1
• Component/s: SQL
• Labels: correctness pull-request-available
• Assignee: Emil Ejbyfeldt
• Reporter: Emil Ejbyfeldt
• Created: 30/Sep/23 09:31
• Updated: 24/Nov/23 22:28
• Resolved: 05/Oct/23 00:41

Description

When using spark 3.5.0 this code

import org.apache.spark.storage.StorageLevel
spark.createDataset(Seq(1,2,3)).persist(StorageLevel.NONE).count() 

incorrectly returns 0.

Issue Links

• links to: GitHub Pull Request #43188
• links to: GitHub Pull Request #43213