[SOLR-12635] HashQParserPlugin should be run as a post filter cost is not explicitly defined

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Varun Thacker
• Reporter: Varun Thacker
• Created: 07/Aug/18 22:12
• Updated: 21/Nov/19 00:41
• Resolved: 20/Aug/18 21:16

Description

I was doing some performance benchmarking for a user on slow streaming queries

The weird thing was that same streaming expression was fast when we fired it again

We were able to isolate the slowness to hash query parser

Here is the first and second time we fired the query - to simplify things this is for one shard and for the same worker

path=/export params={q=*:*&distrib=false&indent=off&fl=fields&fq=user:1&fq={!hash workers=6 worker=3}&partitionKeys=partitionKey&sort=partitionKey asc&wt=javabin&version=2.2} hits=0 status=0 QTime=6821

path=/export params={q=*:*&distrib=false&indent=off&fl=fields&fq=user:1&fq={!hash workers=6 worker=3}&partitionKeys=partitionKey&sort=partitionKey asc&wt=javabin&version=2.2} hits=0 status=0 QTime=0

Even with hits=0 the first query took 6.8 seconds. The shard has 17m documents 

The second query utilizes the queryResultCache and hence it's lightening fast the second time around.

When we execute the same query and add a cost i.e &fq={!hash workers=6 worker=3 cost=101} the query get's executed as a post filter and even uncashed is super fast.

I created this Jira so that we can always set cost > 100 from the parallel stream.

However I am happy to change the default behaviour for HashQParserPlugin and make it run as a post filter always unless explicitly specified. CollapsingQParserPlugin does this currently to make sure it's run as a post filter by default

public int getCost() {
  return Math.max(super.getCost(), 100);
}

Thoughts anyone? 

 

Issue Links

• is related to: SOLR-12684 Document speed gotchas and partitionKeys usage for ParallelStream