[SPARK-23223] Stacking dataset transforms performs poorly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Assignee: Herman van Hövell
• Reporter: Herman van Hövell
• Created: 26/Jan/18 03:29
• Updated: 12/Dec/22 18:10
• Resolved: 29/Jan/18 17:02

Description

It is a common pattern to apply multiple transforms to a Dataset (using Dataset.withColumn for example. This is currently quite expensive because we run CheckAnalysis on the full plan and create an encoder for each intermediate Dataset.

CheckAnalysis only needs to be run for the newly added plan components, and not for the full plan. The addition of the AnalysisBarrier created this issue.

Issue Links

• relates to: SPARK-17006 WithColumn Performance Degrades with Number of Invocations
• links to: [Github] Pull Request #20402 (hvanhovell)