Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.3.0
-
None
-
None
Description
The motivation of adding DSV2 filters:
The values in V1 filters are Scala types. When translating catalyst Expression to V1 filers, we have to call convertToScala to convert from Catalyst types used internally in rows to standard Scala types, and later convert Scala types back to Catalyst types. This is very inefficient. In V2 filters, we use Expression for filter values, so the conversion from Catalyst types to Scala types and Scala types back to Catalyst types are avoided.
Attachments
1.
|
Add DSV2 filters |
|
Resolved | Huaxin Gao |
2.
|
Use V2 Filter in V2 file source |
|
In Progress | Unassigned |
3.
|
Add interface SupportsPushDownV2Filters |
|
Resolved | Huaxin Gao |
4.
|
Return Array of Predicate for SupportsPushDownCatalystFilters.pushedFilters |
|
Resolved | Huaxin Gao |
5.
|
V2 Filter to Parquet Predicate support |
|
Open | Unassigned |
6.
|
V2 Filter to ORC Predicate support |
|
Open | Unassigned |
7.
|
Add DS V2 Filter to V1 Filter conversion |
|
Resolved | Huaxin Gao |
8.
|
Use V2 Filter in SupportsDelete |
|
Resolved | Huaxin Gao |
9.
|
Use V2 Filter in SupportsRuntimeFiltering |
|
Resolved | Huaxin Gao |
10.
|
Use V2 Filter in SupportsOverwrite |
|
Resolved | Huaxin Gao |