[SPARK-21510] Add isMaterialized() and eager persist() to Dataset APIs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Xiao Li
• Reporter: Xiao Li
• Created: 23/Jul/17 04:21
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:44

Description

Currently, when using Spark, the beginners do not realize our persist API is lazy. They do not know what is the most efficient way to materialize it. Sometimes, they just use collect(), which is very expensive when the data set is big.

In addition, we also need another API to verify whether the Dataset has been cached and materialized.

Issue Links

• links to: [Github] Pull Request #18717 (gatorsmile)
• links to: GitHub Pull Request #18717