[SPARK-45889] Implement push-down filter with partition ID and grouping key (if possible) for state data source reader

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 10/Nov/23 14:47
• Updated: 10/Nov/23 14:50

Description

If the query filters the state data via partition ID, it is a good chance for state data source to avoid spinning all state store instances and wasting resource. We can spin state store instances for only necessary partitions.

Same thing applies to grouping keys, although the criteria on distribution is bound to the operator rather than the key in state store, hence it could be very tricky unless we can follow the same criteria on distribution for the operator.

Issue Links

• depends upon: SPARK-45511 SPIP: State Data Source - Reader