[SPARK-49070] TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL, Structured Streaming
• Labels: pull-request-available
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 31/Jul/24 06:09
• Updated: 01/Aug/24 19:32
• Resolved: 01/Aug/24 19:27

Description

SPARK-47363 added the support for users to provide initial state for streaming query. Such query operators like `TransformWithStateExec` might have `hasInitialState` as false which means the initial state related parameters are not used. But when query planner applies rules on the query, it will still apply on the initial state query plan. When `hasInitialState` is false, some related parameters like `initialStateGroupingAttrs` are invalid and some rules will use these invalid parameters to transform the initial state query plan.

For example, `EnsureRequirements` may apply invalid Sort and Exchange on the initial query plan. We encountered these invalid query plan in our extension rules.

Issue Links

• links to: GitHub Pull Request #47546