[SPARK-44464] Fix applyInPandasWithStatePythonRunner to output rows that have Null as first column value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.3
• Fix Version/s: 3.4.2, 3.5.0
• Component/s: Structured Streaming
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 17/Jul/23 20:00
• Updated: 21/Jul/23 05:11
• Resolved: 19/Jul/23 01:19

Description

The current implementation of ApplyInPandasWithStatePythonRunner cannot deal with outputs where the first column of the row is null, as it cannot distinguish the case where the column is null, or the field is filled as the number of data records are smaller than state records. It causes incorrect results for the former case.