[SPARK-41261] applyInPandasWithState can produce incorrect key value in user function for timed out state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 25/Nov/22 07:01
• Updated: 12/Dec/22 18:10
• Resolved: 27/Nov/22 02:01

Description

We observed the issue that user function retrieves incorrect key in user function for timed out state. After RCA we figured out this could happen when the columns of grouping keys are not placed sequentially at earliest place.

Issue Links

• links to: [Github] Pull Request #38798 (HeartSaVioR)