Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
From the conversation on dev@pig:
"Partition pruning for ORC is not addressed in PIG-3558. We will need
to do partition pruning for both ORC and Parquet in a new ticket.
Curently there is no interface to deal with this kind of pushdown
(LoadMetadata.setPartitionFilter push the filter to loader, but remove
the filter statement, for ORC/Parquet, filter is a hint, and we need
to do the filter again in Pig even it is pushed to loader), we will
need to define a new interface for that. You are welcome to initiate
the work. I know Aniket is also interested in doing that, so be sure
the talk with him about this work.
Thanks,
Daniel
On Mon, Feb 10, 2014 at 11:42 AM, Andrew Musselman
<andrew.musselman@gmail.com> wrote:
> I had a chat with a couple people last week about a feature request for
> Pig: in a "where" or "filter" clause, when loading an ORC file, to skip
> directly to the right offset instead of scanning the whole file."
Attachments
Issue Links
- relates to
-
HIVE-7214 Support predicate pushdown for complex data types in ORCFile
-
- Open
-
1.
|
Predicate pushdown for ORC |
|
Closed | Rohini Palaniswamy |
2.
|
Predicate pushdown for Parquet |
|
Resolved | Nándor Kollár |
3.
|
Predicate pushdown to support removing filters from pig plan |
|
Open | Unassigned |
4.
|
Predicate pushdown to support complex data types |
|
Open | Unassigned |
5.
|
Collapse multiple OR conditions to IN and BETWEEN |
|
Open | Unassigned |
6.
|
Implement predicate pushdown evaluation in OrcStorage |
|
Open | Unassigned |
7.
|
Adding e2e tests and several improvements for Orc predicate pushdown |
|
Closed | Daniel Dai |
8.
|
Make LoadPredicatePushdown InterfaceAudience.Private till PIG-4093 |
|
Closed | Rohini Palaniswamy |