[SPARK-39748] Include the origin logical plan for LogicalRDD if it comes from DataFrame

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL, Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 12/Jul/22 01:45
• Updated: 14/Jul/22 03:47
• Resolved: 12/Jul/22 09:00

Description

When Spark converts the DataFrame to LogicalRDD for some reason (e.g. foreachBatch sink), Spark just picks the RDD from the origin DataFrame and discards the (logical/physical) plan.

The origin logical plan can be useful for several use cases, including:

1. wants to connect the overall logical plan into one
2. inherits plan stats from origin logical plan

Issue Links

• links to: [Github] Pull Request #37161 (HeartSaVioR)
• links to: [Github] Pull Request #37167 (HeartSaVioR)
• links to: [Github] Pull Request #37187 (HeartSaVioR)