First draft. The patch doesn't include test yet.

I added optimizeForNgram() methods in the patch that will remove "redundant" terms and positions.

Optimizing PhraseQuery will change score.

I added test code.

The result of speed up is:

Hi Koji, I wonder if instead it would be cleaner as a subclass of PhraseQuery (NGramPhraseQuery or similar),
that rewrites to the (possibly optimized) PhraseQuery in rewrite(). For example, it would build an optimized
PhraseQuery when slop = 0, and there are enough terms to optimize, otherwise it would build a "normal" phrasequery.

Then the optimization would be easy to apply, the user just uses NGramPhraseQuery instead of PhraseQuery.
for example, from QueryParser:

  @Override
  protected PhraseQuery newPhraseQuery() {
    return new NGramPhraseQuery();
  }

I like the idea of introducing the newly created class! Here is the new patch.

I think I like it better too... though I wonder if its possible to keep the original NGramPhraseQuery unmodified?
this way its not changed by Query.rewrite(), and if a user reuses the query (which we document they can do), they could then call add() again and everything works.

Also, somewhat related to the issue might be SOLR-2660. We don't have to commit that patch, but we could separate
out the queryparser refactoring to make it easier for such an optimization to be "automatic" in solr, because it allows
SolrQueryParser to delegate creation of Phrase/MultiPhraseQuery to the FieldType.

I think I like it better too... though I wonder if its possible to keep the original NGramPhraseQuery unmodified?
this way its not changed by Query.rewrite(), and if a user reuses the query (which we document they can do), they could then call add() again and everything works.

I wonder it that too. Here is the new patch. This time I added assertSame()/NotSame() to check the rewritten Query to test code.

For automatic in Solr, I wonder if we could move the feature to n-gram tokenizers, and we could have something like:

<fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100"
           autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <tokenizer class="solr.CJKTokenizerFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.CJKTokenizerFactory" optimizePhraseQuery="true"/>
  </analyzer>
</fieldType>

Well if we apply the refactoring part of SOLR-2660 (we can split out into a separate issue), we could add such a thing as an attribute to the fieldType?

I like the way your patch looks now! A couple more questions:

• doesn't the optimization also apply to MultiPhraseQuery? If so, NGramPhraseQuery could extend MultiPhraseQuery and just rewrite to the correct one (MultiPhrase or Phrase depending upon the situation after optimization)
• what about hashCode/equals? Although the same results will be returned, scoring will differ, maybe it NGramPhraseQuery should implement these?

I'm not sure it could apply MutiPhraseQuery. Let me take more time.

Considering hashCode/equals is good point. I'll see.

New patch. I added equals/hashCode in the patch.

I think it is too complex to apply optimization to MultiPhraseQuery, so I'd like to stick with PhraseQuery in the patch.

I think I agree Koji, the patch looks good.

Though we should be able to keep PhraseQuery's internal members 'private' since NGramPhraseQuery now uses getter methods to access positions, terms, slop ?

Patch looks great! What a nice opto

Thank you for your continuous review the patch, Robert! Here is the new patch. Now I don't touch the existing PhraseQuery as I use getter methods.

I think this is ready to commit.

trunk: Committed revision 1170586.
3x: Committed revision 1170593.

Bulk close after release of 3.5

Is this automatic in SOLR? Or do we need to add a feature to support his in SOLR?

Is this automatic in SOLR?

No. I've opened SOLR-3055.