Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.5.0
-
None
Description
For EasyGroupScan that handles Json and Text formats, we are not getting the directory partition filters pushed down into the Scan. One part of the reason is that the EasyGroupScan digest for a new scan that has a smaller selection of files is currently the same as the previous scan, so the planner ignores the new scan.
A second related issue is that the Filter can appear above either a Scan or a Project, so we need to be able to handle both cases during pushdown.
Uploaded a patch for this issue. Also added a whole bunch of Junit tests for push partition filter on all 3 formats: parquet, json and csv.