The way to generate a query based on the token stream of the query string is the responsibility of the query parser. For example, QueryParserBase has the setEnablePositionIncrements method http://lucene.apache.org/core/4_4_0/queryparser/org/apache/lucene/queryparser/classic/QueryParserBase.html#setEnablePositionIncrements(boolean) which can be used to ignore holes in the token stream (which seems to be what you are looking for).

I do index
twitter.com/testuser
then search for
http://www.twitter.com/testuser

These are in stopwords filter:
http
https
www

No results.

Hi, how we're now supposed to fix this?
http://stackoverflow.com/questions/18668376/solr-4-4-stopfilterfactory-and-enablepositionincrements

There should be an option in AnalyzingLookupFactory and FuzzyLookupFactory to setPreservePositionIncrements then.

There should be an option in AnalyzingLookupFactory and FuzzyLookupFactory to setPreservePositionIncrements then.

Bulk close resolved 4.4 issues

Thank you Uwe!

[trunk commit] jpountz
http://svn.apache.org/viewvc?view=revision&revision=1479171

LUCENE-4963: Completely remove deprecated options in 5.0.

[branch_4x commit] jpountz
http://svn.apache.org/viewvc?view=revision&revision=1479151

LUCENE-4963: Deprecate broken TokenFilter options (merged from r1479148).

[trunk commit] jpountz
http://svn.apache.org/viewvc?view=revision&revision=1479148

LUCENE-4963: Deprecate broken TokenFilter options.

I'll commit this soon unless someone objects.

Yeah, I would do it the same way. First deprecate and commit to both branches. Then remove stuff in trunk only.

Hi Uwe, thanks for doing the review! The patch applies to trunk and I plan to remove deprecations in a second step. Is it OK with you?

Hi Adrien,
thanks for doing the work! I think on the first look its fine! I will now verify the changes for each TokenFilter... The patch is for 4.x? In trunk you will remove the deprecations in a second step to enable merging without extra work?

Thanks Uwe for the advice. Here is a first patch:

• Deprecate constructors that expose broken options and make them throw an IllegalArgumentException when the lucene match version is >= 4.4

• Remove the same constructors from TestRandomChains' exclusion list.

• Since enablePositionIncrements=true was used by the Analyzing and Fuzzy suggesters to ignore position holes, I had to make it an option in the suggesters themselves instead of the token streams.

• More documentation in the oal.analysis package: PositionLengthAttribute and guidelines on writing non-corrupt token streams.

Hi,
+1 to deprecate the constuctors taking those arguments and make sure the alternatives already implement the correct behaviour.
Unfortunately we may not be able to completely remove that code in 5.x (we can remove the ctors, yes), because matchVersion sometimes implies the wrong behaviour. We nuked almost all of them, we should just check this.