[SPARK-10899] Support JDBC pushdown for additional commands

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.5.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Victor May
• Created: 01/Oct/15 14:31
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/16 12:37

Description

Spark DataFrames currently support predicate push-down with JDBC sources but term predicate is used in a strict SQL meaning. It means it covers only WHERE clause. Moreover it looks like it is limited to the logical conjunction (no IN and OR I am afraid) and simple predicates.

This creates a situation where a simple query such as "select * from table limit 100" could easily result in the database being overloaded when the table is large.

This feature request is to expand the support for push-downs to additional SQL commands:
-LIMIT
-WHERE IN
-WHERE NOT IN
-GROUP BY

Issue Links

• duplicates: SPARK-12126 JDBC datasource processes filters only commonly pushed down.
• relates to: SPARK-12449 Pushing down arbitrary logical plans to data sources