Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38852

Better Data Source V2 operator pushdown framework

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • SQL
    • None

    Description

      Currently, Spark supports push down Filters and Aggregates to data source.
      However, the Data Source V2 operator pushdown framework has the following shortcomings:

      1. Only simple filter and aggregate are supported, which makes it impossible to apply in most scenarios
      2. The incompatibility of SQL syntax makes it impossible to apply in most scenarios
      3. Aggregate push down does not support multiple partitions of data sources
      4. Spark's additional aggregate will cause some overhead
      5. Limit push down is not supported
      6. Top n push down is not supported
      7. Aggregate push down does not support group by expressions
      8. Aggregate push down does not support not use aggregate functions
      9. Offset push down is not supported
      10. Paging push down is not supported
      11. UDF/UDAF push down is not supported

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              beliefer Jiaan Geng
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: