[SPARK-43183] Move update event on idleness in streaming query listener to separate callback method

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 19/Apr/23 09:45
• Updated: 20/Apr/23 08:21
• Resolved: 20/Apr/23 08:21

Description

People has been having a lot of confusions about update event on idleness; it’s not only the matter of understanding but also comes up with various types of complaints. For example, since we give the latest batch ID for update event on idleness, if the listener implementation blindly performs upsert based on batch ID, they are in risk to lose metrics.

This also complicates the logic because we have to memorize the execution for the previous batch, which is arguably not necessary.

Because of this, we’d be better to move the idle event out of progress update event and have separate callback method for this.