[#LUCENE-966] A faster JFlex-based replacement for StandardAnalyzer

Lucene - Java

A faster JFlex-based replacement for StandardAnalyzer

Created: 26/Jul/07 01:04 PM Updated: 25/Jan/08 03:24 AM

Return to search

Component/s:

Analysis

Affects Version/s:

None

Fix Version/s:

2.3

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
AnalyzerBenchmark.java	2007-07-26 01:15 PM	Stanislaw Osinski	4 kB
jflex-analyzer-patch.txt	2007-07-26 01:12 PM	Stanislaw Osinski	52 kB
jflex-analyzer-r560135-patch.txt	2007-07-27 08:05 AM	Stanislaw Osinski	67 kB
jflex-analyzer-r561292-patch.txt	2007-07-31 10:05 AM	Stanislaw Osinski	143 kB
jflex-analyzer-r561693-compatibility.txt	2007-08-01 08:33 AM	Stanislaw Osinski	132 kB
jflex-analyzer-r562378-patch-nodup.txt	2007-08-03 12:37 PM	Stanislaw Osinski	133 kB
jflex-analyzer-r562378-patch.txt	2007-08-03 08:19 AM	Stanislaw Osinski	236 kB

Lucene Fields:	New, Patch Available
Resolution Date:	08/Aug/07 10:29 PM

Description

« Hide

JFlex (http://www.jflex.de/) can be used to generate a faster (up to several times) replacement for StandardAnalyzer. Will add a patch and a simple benchmark code in a while.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#564036	Wed Aug 08 22:26:44 UTC 2007	mikemccand	~~LUCENE-966~~: sizable (~6X faster) speedups to StandardTokenizer by using JFlex instead of JavaCC
				Files Changed
				ADD /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex MODIFY /lucene/java/trunk/contrib/miscellaneous/src/test/org/apache/lucene/misc/ChainedFilterTest.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/package.html DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj ADD /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/Token.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/ParseException.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/CharStream.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardFilter.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerConstants.java MODIFY /lucene/java/trunk/build.xml DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/FastCharStream.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java DEL /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/TokenMgrError.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/common-build.xml

Repository	Revision	Date	User	Message
ASF	#616248	Tue Jan 29 10:51:44 UTC 2008	mikemccand	~~LUCENE-1150~~: make StandardAnalyzer tokenizer constants public again (public access was accidentally removed with ~~LUCENE-966~~)
				Files Changed
				MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java MODIFY /lucene/java/trunk/contrib/wikipedia/src/java/org/apache/lucene/wikipedia/analysis/WikipediaTokenizer.java MODIFY /lucene/java/trunk/contrib/wikipedia/src/java/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerImpl.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/analysis/TestAnalyzers.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java MODIFY /lucene/java/trunk/contrib/wikipedia/src/java/org/apache/lucene/wikipedia/analysis/WikipediaTokenizerImpl.jflex