Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-577

Allow row-level filtering

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.7.0
    • 1.7.0
    • None

    Description

      Currently, ORC filters at three levels:

      • File level
      • Stripe (64 to 256mb) level
      • Row group (10k row) level

      The filters are specified as Sargs (Search Arguments), which have a relatively small vocabulary. Furthermore, they only filter sets of rows if they can guarantee that none of the rows can pass the filter.

      There are some use cases where the user needs to read a subset of the columns and apply more detailed row level filters. I'd suggest that we add a new method in Reader.Options

      setRowFilter(String[] filterColumnNames, Consumer<VectorizedRowBatch> filterCallback))

      Where the columns named in columnNames are read expanded first, then the filter is run and the rest of the data is read only if the predicate returns true.

      Attachments

        1. RowFilterBenchTimestamp.out
          0.9 kB
          Panagiotis Garefalakis
        2. RowFilterBenchString.out
          0.8 kB
          Panagiotis Garefalakis
        3. RowFilterBenchBoolean.out
          0.9 kB
          Panagiotis Garefalakis
        4. RowFilterBenchDecimal.out
          2 kB
          Panagiotis Garefalakis
        5. RowFilterBenchDouble.out
          0.9 kB
          Panagiotis Garefalakis

        Issue Links

          is a parent of

          Improvement - An improvement or enhancement to an existing feature or task. ORC-597 Row-level filtering bench

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. ORC-619 Row-level filtering support for nested types

          • Major - Major loss of function.
          • Open
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. ORC-620 Modify the row filter API to use BiFunction

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-22731 MJ probe decode with row-level filtering

          • Major - Major loss of function.
          • Patch Available
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. ORC-593 Allow row-level Skipping

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. ORC-744 LazyIO of non-filter columns

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #475

          Activity

            People

              pgaref Panagiotis Garefalakis
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m