Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.3.0
-
None
-
None
Description
The motivation of adding DSV2 filters:
The values in V1 filters are Scala types. When translating catalyst Expression to V1 filers, we have to call convertToScala to convert from Catalyst types used internally in rows to standard Scala types, and later convert Scala types back to Catalyst types. This is very inefficient. In V2 filters, we use Expression for filter values, so the conversion from Catalyst types to Scala types and Scala types back to Catalyst types are avoided.
Attachments
1.
|
Use V2 Filter in V2 file source |
|
In Progress | Unassigned |
2.
|
V2 Filter to Parquet Predicate support |
|
Open | Unassigned |
3.
|
V2 Filter to ORC Predicate support |
|
Open | Unassigned |