[SPARK-48597] Distinguish the streaming nodes from the text representation of logical plan

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 12/Jun/24 07:33
• Updated: 17/Jun/24 00:20
• Resolved: 17/Jun/24 00:20

Description

We had a hard time to figure out whether the nodes are streaming or not, when we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 .

Plan text for logical plan does not show the property of isStreaming, hence we had to speculate the value based on other context. In addition, even though the type of leaf node is explicitly meant to be streaming which enables us to track down the isStreaming for certain subtree, the plan could be very long and it’s a non-trivial effort to trace down to the leaf nodes. Also, if the leaf nodes are skipped on the representation due to the size, there is no way to get the information of isStreaming.

We propose to introduce a marker of the representation for streaming, which will be shown in the text logical plan. There is no concept of "isStreaming" in physical plan, so the change only needs to happen in logical plan.

Issue Links

• links to: GitHub Pull Request #46953