[SPARK-12449] Pushing down arbitrary logical plans to data sources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Stephan Kessler
• Created: 21/Dec/15 12:43
• Updated: 08/Oct/19 05:44
• Resolved: 08/Oct/19 05:44

Description

With the help of the DataSource API we can pull data from external sources for processing. Implementing interfaces such as PrunedFilteredScan allows to push down filters and projects pruning unnecessary fields and rows directly in the data source.

However, data sources such as SQL Engines are capable of doing even more preprocessing, e.g., evaluating aggregates. This is beneficial because it would reduce the amount of data transferred from the source to Spark. The existing interfaces do not allow such kind of processing in the source.

We would propose to add a new interface CatalystSource that allows to defer the processing of arbitrary logical plans to the data source. We have already shown the details at the Spark Summit 2015 Europe https://spark-summit.org/eu-2015/events/the-pushdown-of-everything/

I will add a design document explaining details.

Issue Links

• is duplicated by: SPARK-19655 select count(*) , requests 1 for each row
• is duplicated by: SPARK-20259 Support push down join optimizations in DataFrameReader when loading from JDBC
• is related to: SPARK-9182 filter and groupBy on DataFrames are not passed through to jdbc source
• is related to: SPARK-10899 Support JDBC pushdown for additional commands
• is related to: SPARK-12126 JDBC datasource processes filters only commonly pushed down.
• is related to: SPARK-12506 Push down WHERE clause arithmetic operator to JDBC layer
• is related to: SPARK-12686 Support group-by push down into data sources