Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-384

Add Dictionary Based Filtering to Filter2 API

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      Dictionary based predicate evaluation show very significant gains for certain cases like the following:

      • Sparse values in dictionary encoded columns
      • Data staged sorted by fields used in predicate evaluation

      This would be good to introduce into the filter2 API.

        Issue Links

          Activity

          Hide
          dweeks-netflix Daniel Weeks added a comment -

          https://github.com/apache/parquet-mr/pull/286

          Alex Levenson Can you take a look at the changes I made to accommodate the dictionary filter?

          Show
          dweeks-netflix Daniel Weeks added a comment - https://github.com/apache/parquet-mr/pull/286 Alex Levenson Can you take a look at the changes I made to accommodate the dictionary filter?
          Hide
          rdblue Ryan Blue added a comment -

          I submitted an updated PR, #330.

          Show
          rdblue Ryan Blue added a comment - I submitted an updated PR, #330.
          Hide
          dweeks-netflix Daniel Weeks added a comment - - edited

          This now includes the api for reading dictionaries and supersedes PARQUET-374.

          Show
          dweeks-netflix Daniel Weeks added a comment - - edited This now includes the api for reading dictionaries and supersedes PARQUET-374 .
          Hide
          rdblue Ryan Blue added a comment -

          Merged #330. Thanks for reviewing, Dan!

          Show
          rdblue Ryan Blue added a comment - Merged #330. Thanks for reviewing, Dan!

            People

            • Assignee:
              dweeks-netflix Daniel Weeks
              Reporter:
              dweeks-netflix Daniel Weeks
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development