[SPARK-47305] PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 4.0.0, 3.5.1
• Fix Version/s: 4.0.0, 3.5.2, 3.4.3
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 06/Mar/24 12:59
• Updated: 08/Mar/24 08:46
• Resolved: 07/Mar/24 06:12

Description

This seems to be a very old bug in optimizer. Related ticket:  https://issues.apache.org/jira/browse/SPARK-21765

When filter is evaluated to be always false, PruneFilters replaces the filter with empty LocalRelation, which effectively prunes filter. The logic cares about migration of the isStreaming flag, but incorrectly migrated in some case, via picking up the value of isStreaming flag from root node rather than filter (or child).

isStreaming flag is true if the value of isStreaming flag from any of children is true. Flipping the coin, some children might have isStreaming flag as "false". If the filter being pruned is a descendant to such children (in other word, ancestor of streaming node), LocalRelation is incorrectly tagged as streaming where it should be batch.

Issue Links

• links to: GitHub Pull Request #45406
• links to: GitHub Pull Request #45428