Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-652

support block-based filtering within RFile

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: tserver
    • Labels:
      None

      Description

      If we keep some stats about what is in an RFile block, we might be able to efficiently [O(log N)], with high probability, implement filters that currently require linear table scans. Two use cases of this include timestamp range filtering (i.e. give me everything from last Tuesday) and cell-level security filtering (i.e. give me everything that I can see with my authorizations).

      For the timestamp range filter, we can keep minimum and maximum timestamps across all keys used in a block within the index entry for that block. For the cell-level security filter, we can keep an aggregate label. This could be done using a simplified disjunction of all of the labels in the block. The extra block statistics information can propagate up the index hierarchy as well, giving nice performance characteristics for finding the next matching entry in a file.

      In general, this is a heuristic technique that is good if data tends to naturally cluster in blocks with respect to the way it is queried. Testing its efficacy will require closely emulating real-world use cases – tests like the continuous ingest test will not be sufficient. We will have to test for a few things:

      1. The cost for storing the extra stats in the index are not too expensive.
      2. The performance benefit for common use cases is significant.
      3. We shouldn't introduce any unacceptable worst-case behavior, like bloating the index to ridiculous proportions for any data set.

      Eventually this will all need to be exposed through the Iterator API to be useful, which will be another ticket.

        Attachments

        1. ACCUMULO-652-patches.tar.gz
          36 kB
          Christopher Tubbs

          Issue Links

            Activity

              People

              • Assignee:
                afuchs Adam Fuchs
                Reporter:
                afuchs Adam Fuchs
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m