[SPARK-48597] Distinguish the streaming nodes from the text representation of logical plan - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 4.0.0
Fix Version/s: 4.0.0
Component/s: SQL
Labels:
- pull-request-available

Description

We had a hard time to figure out whether the nodes are streaming or not, when we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 .

Plan text for logical plan does not show the property of isStreaming, hence we had to speculate the value based on other context. In addition, even though the type of leaf node is explicitly meant to be streaming which enables us to track down the isStreaming for certain subtree, the plan could be very long and it’s a non-trivial effort to trace down to the leaf nodes. Also, if the leaf nodes are skipped on the representation due to the size, there is no way to get the information of isStreaming.

We propose to introduce a marker of the representation for streaming, which will be shown in the text logical plan. There is no concept of "isStreaming" in physical plan, so the change only needs to happen in logical plan.