Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-10103

QueryCache not estimating query size properly

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • None
    • core/search
    • None
    • New

    Description

      QueryCache seems estimating the cached query size using a constant, it will cause OOM error in some extreme cases where queries cached will use far more memories than assumed. (The default QueryCache tries to use only 5% of heap) One example of such memory-eating query is AutomatonQuery, it will each carry a RunAutomaton , which consumes a good amount of memory in exchange for the speed.

      On the other hand, we actually have a good implementation of Accountable interface for AutomatonQuery (though it will become a bit more complicated later since this query will eventually be rewritten to something else), so maybe QueryCache could use those estimation directly (using an instanceof check)? Or moreover we could make all Query implement {Accountable}}, and maybe the default implementation could just be returning the current constant we're using, and only override the method of the potential troublesome queries?

      Attachments

        1. query_cache_error_demo.patch
          2 kB
          Patrick Zhai

        Activity

          People

            Unassigned Unassigned
            zhai7631 Patrick Zhai
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m

                Slack

                  Issue deployment