[#LUCENE-967] Add "tokenize documents only" task to contrib/benchmark

Lucene - Java

Add "tokenize documents only" task to contrib/benchmark

Created: 26/Jul/07 06:08 PM Updated: 25/Jan/08 03:24 AM

Return to search

Component/s:

contrib/benchmark

Affects Version/s:

2.3

Fix Version/s:

2.3

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
LUCENE-967.patch	2007-07-26 06:09 PM	Michael McCandless	11 kB
LUCENE-967.take2.patch	2007-07-29 12:59 AM	Michael McCandless	12 kB
LUCENE-967.take3.patch	2007-08-01 12:16 PM	Michael McCandless	14 kB

Lucene Fields:	Patch Available, New
Resolution Date:	01/Aug/07 06:55 PM

Description

« Hide

I've been looking at performance improvements to tokenization by
re-using Tokens, and to help benchmark my changes I've added a new
task called ReadTokens that just steps through all fields in a
document, gets a TokenStream, and reads all the tokens out of it.

EG this alg just reads all Tokens for all docs in Reuters collection:

doc.maker=org.apache.lucene.benchmark.byTask.feeds.ReutersDocMaker
doc.maker.forever=false
{ReadTokens > : *

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Michael McCandless made changes - 26/Jul/07 06:09 PM

Field	Original Value	New Value
Attachment		LUCENE-967.patch [ 12362636 ]

Michael McCandless made changes - 26/Jul/07 06:10 PM

Status

Open [ 1 ]

In Progress [ 3 ]

Michael McCandless made changes - 26/Jul/07 06:10 PM

Lucene Fields

[New]

[New, Patch Available]

Michael McCandless made changes - 29/Jul/07 12:59 AM

Attachment

LUCENE-967.take2.patch [ 12362728 ]

Michael McCandless made changes - 01/Aug/07 12:16 PM

Attachment

LUCENE-967.take3.patch [ 12362968 ]

Michael McCandless made changes - 01/Aug/07 06:55 PM

Lucene Fields	[Patch Available, New]	[New, Patch Available]
Status	In Progress [ 3 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Michael Busch made changes - 25/Jan/08 03:24 AM

Status

Resolved [ 5 ]

Closed [ 6 ]