Solr
  1. Solr
  2. SOLR-3333

Create an option that allows a query to be cached, but not used for warming

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.5, 4.0-ALPHA
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The application that uses my Solr install builds complex filter queries for employees because they have access to everything, whereas most users have access to a small subset.

      Because of this, autowarming on the filterCache can take 30-60 seconds even though autoWarm is set to just 4 queries.

      If we had a way (probably a localparam) to tell Solr to not use those filters when autowarming, but to go ahead and put them in the filterCache and use them until there's a new commit, that would eliminate this problem. Employees might have their queries take longer, but regular users would not be affected.

        Activity

        Hide
        Shawn Heisey added a comment -

        I don't think I can implement this. My knowledge of Solr internals simply isn't strong enough.

        Show
        Shawn Heisey added a comment - I don't think I can implement this. My knowledge of Solr internals simply isn't strong enough.
        Hide
        Erick Erickson added a comment -

        Are there other auto-warming queries you want to have done? Because it almost sounds like you just want to turn off autowarming in the filter cache.

        Or if they're unlikely to be re-used anyway, would it work to set cache=false on the original fq?

        Show
        Erick Erickson added a comment - Are there other auto-warming queries you want to have done? Because it almost sounds like you just want to turn off autowarming in the filter cache. Or if they're unlikely to be re-used anyway, would it work to set cache=false on the original fq?
        Hide
        Shawn Heisey added a comment -

        I would like to have our application code tag those nasty employee filters with something that makes them ineligible for autowarming, but still eligible for caching, which would keep them around until the next commit. I am pretty sure our code is capable of knowing that the user is a special user, typically admin or system.

        An update cycle runs once a minute for the index as a whole, but changes are tracked on a per-shard basis. Commits on each shard are only done if something on that particular shard actually changes. The large shards where this is a problem typically go several minutes between commits, and that might extend to an hour or more.

        I will talk to our developers about using the cache=false localparam for now, but I am hoping for the ability to use the cache for those nasty filters but not include them for warming. Having recently toyed with the cache code (SOLR-2906), I know this may not be trivial.

        Show
        Shawn Heisey added a comment - I would like to have our application code tag those nasty employee filters with something that makes them ineligible for autowarming, but still eligible for caching, which would keep them around until the next commit. I am pretty sure our code is capable of knowing that the user is a special user, typically admin or system. An update cycle runs once a minute for the index as a whole, but changes are tracked on a per-shard basis. Commits on each shard are only done if something on that particular shard actually changes. The large shards where this is a problem typically go several minutes between commits, and that might extend to an hour or more. I will talk to our developers about using the cache=false localparam for now, but I am hoping for the ability to use the cache for those nasty filters but not include them for warming. Having recently toyed with the cache code ( SOLR-2906 ), I know this may not be trivial.
        Hide
        Shawn Heisey added a comment -

        I never actually answered your first question. Yes, I do want most entries in the filter cache to be usable for autowarming. Most users have relatively few boolean clauses in their filter queries. Employees are the common exception. We get a few hundred boolean clauses in ours. Plans are being discussed to greatly reduce that, but I'm not sure we'll ever get away from it entirely.

        Show
        Shawn Heisey added a comment - I never actually answered your first question. Yes, I do want most entries in the filter cache to be usable for autowarming. Most users have relatively few boolean clauses in their filter queries. Employees are the common exception. We get a few hundred boolean clauses in ours. Plans are being discussed to greatly reduce that, but I'm not sure we'll ever get away from it entirely.
        Hide
        Shawn Heisey added a comment -

        I just thought of a localparam syntax for this:

        {!cache=nowarm}
        Show
        Shawn Heisey added a comment - I just thought of a localparam syntax for this: {!cache=nowarm}
        Hide
        Shawn Heisey added a comment -

        I hope someone can fix this, but I know that at this time it's not something I can tackle without generous hand-holding. If there are no takers soon, I'll go ahead and close the issue.

        This is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean

        Show
        Shawn Heisey added a comment - I hope someone can fix this, but I know that at this time it's not something I can tackle without generous hand-holding. If there are no takers soon, I'll go ahead and close the issue. This is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean

          People

          • Assignee:
            Unassigned
            Reporter:
            Shawn Heisey
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development