Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.1, 4.0-ALPHA
-
Component/s: Schema and Analysis
-
Labels:None
Description
See LUCENE-1370
Issue Links
- relates to
-
LUCENE-1370
Add ShingleFilter option to output unigrams if no shingles can be generated
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Tom,
The Lucene half of this patch pair adds unit tests to src/test/org/apache/lucene/analysis/shingle/ShingleFilterTest.java. Do those tests pass when you run them on your custom lucene build, after applying LUCENE-1370? (cd to the top-level of lucene and then run "ant test -Dtestcase=ShingleFilterTest".) I didn't add any tests for the Solr half of the patch pair, but I also don't know how you would test it in a productive manner.
Hi Chris,
Thanks for your kind reply. The lucene unit tests passed. It turns out that we had a configuration error that left an unpatched version of ShingleFilter on the classpath when Solr started up. Once we made sure that the patched version was loading, everything has been working just fine.
Tom
Updated patch to reflect changed option name from LUCENE-1370 (outputUnigramIfNoNgram -> outputUnigramsIfNoShingles. Added a simple test to TestShingleFilterFactory.java for the single input token case. Added a solr/CHANGES.txt entry.
Unless there are objections, I will commit this in a couple of days, after LUCENE-1370 has been committed.
Committed: trunk revision 1006191, branch_3x revision 1006199
Bulk close for 3.1.0 release
I applied both this and
LUCENE-1370and there seems to be some problem with passing arguments from the ShingleFilterFactory to the ShingleFilter. The admin analyzer says that outputUnigramIfNoNgram=trueorg.apache.solr.analysis.ShingleFilterFactory
{outputUnigrams=false, outputUnigramIfNoNgram=true}However, this does not seem to be getting set within the ShingleFilter and the admin analyzer shows nothing coming out of the ShingleFilterFactory when analyzing a query with a single word.
when using the admin interface to query a single word, I also get no results.
If I hack the patch by always setting outputUnigramsIfNoNgrams to true, everything works fine.
(see below)
If I am missing something or obviously doing something wrong, please let me know. In the meantime I will try to write a unit test and track down the problem. Is there an already existing unit test I could use as a model?
Tom Burton-West
------------------------------------------------------
Hack
public void init(Map<String, String> args)
{ super.init(args); maxShingleSize = getInt("maxShingleSize", ShingleFilter.DEFAULT_MAX_SHINGLE_SIZE); outputUnigrams = getBoolean("outputUnigrams", true); outputUnigramIfNoNgrams = true; /** tbw lets always set it to true above * comment out the original code below getBoolean("outputUnigramIfNoNgram", false); **/ }