[SPARK-38852] Better Data Source V2 operator pushdown framework

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Epic Link: More comprehensive DSV2 push down capabilities
• Assignee: Unassigned
• Reporter: Jiaan Geng
• Created: 11/Apr/22 06:43
• Updated: 15/Sep/23 17:19

Description

Currently, Spark supports push down Filters and Aggregates to data source.
However, the Data Source V2 operator pushdown framework has the following shortcomings:

1. Only simple filter and aggregate are supported, which makes it impossible to apply in most scenarios
2. The incompatibility of SQL syntax makes it impossible to apply in most scenarios
3. Aggregate push down does not support multiple partitions of data sources
4. Spark's additional aggregate will cause some overhead
5. Limit push down is not supported
6. Top n push down is not supported
7. Aggregate push down does not support group by expressions
8. Aggregate push down does not support not use aggregate functions
9. Offset push down is not supported
10. Paging push down is not supported
11. UDF/UDAF push down is not supported

Issue Links

• duplicates: SPARK-36695 Allow passing V2 functions to data sources via V2 filters

Sub-Tasks

• DS V2 supports push down collection functions: Open
• Translate common used aggregate functions for pushdown (non ANSI): Open