[SPARK-31137] Opportunity to simplify execution plan when passing empty dataframes to subtract()

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Do
• Affects Version/s: 2.4.5
• Component/s: PySpark, SQL
• Assignee: Unassigned
• Reporter: S Daniel Zafar
• Created: 12/Mar/20 19:40
• Updated: 13/Mar/20 16:02
• Resolved: 13/Mar/20 16:02

Description

Execution plans are similar when passing an empty versus non-empty DataFrame to pyspark's subtract call.

df.subtract(regDf)

yields the same physical plan as:

df.subtract(emptyDf)

 Since the operation (EXCEPT DISTINCT in Spark SQL) requires a sort on both DataFrames, this can yield some significant performance speed-ups because if the incoming DF is empty no processing should happen.

 

Should be a quick fix for a seasoned commiter.