[SPARK-48481] OptimizeOneRowPlan should not be effective for streaming DataFrame

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.2, 3.4.4
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: Optimizer, Structured Streaming
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 31/May/24 02:51
• Updated: 01/Jun/24 06:36
• Resolved: 01/Jun/24 06:32

Description

This bug is introduce from Apache Spark 3.3.0.

OptimizeOneRowPlan aggressively rewrites operators or removes operators if the rule figures out that the operator has a chance to be optimized with the stats that there will be max 1 row in the input.

This is problematic for streaming, because we aren't seeing the whole data but a part of data in the current microbatch and optimizer is not aware of this.

There are various viable approaches to deal with, but maybe fixing the rule to disable this effectively with streaming DataFrame would have least effect.

(There is a separate wider effort to achieve better stability between QO and streaming. Since it would take a considerable time, we still need point fixes during the time.)

Issue Links

• links to: GitHub Pull Request #46820