[SPARK-39940] Batch query cannot read the updates from streaming query if streaming query writes to the catalog table via DSv1 sink

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.2.3, 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 02/Aug/22 02:13
• Updated: 02/Aug/22 08:56
• Resolved: 02/Aug/22 08:56

Description

(I think this should be ancient issue but there's no good way to list up all affected versions, so I just pick up the recent version in each version line.)

When streaming query writes to catalog table via DSv1 sink, there is no refreshing/invalidation of the destination table, hence querying the destination table with batch query is not guaranteed to read the latest "committed" updates.

Issue Links

• links to: [Github] Pull Request #37368 (HeartSaVioR)
• links to: [Github] Pull Request #37368 (HeartSaVioR)