Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.1.11, 2.2
-
None
-
Normal
-
Patch available
Description
PROBLEM:
The LuceneIndexTransformer optimizes the Lucene index every time you add an entry to the index.
This slows down enormously the indexing with a large index ! If upon every checkin of a document eg,
you use it to update the entry, it will slow down.
Eg. I have a Pentium IV 2.4 Ghz, Lucene index contains 10 000 doc.
Where the index update only takes say 60ms, the optimize that get's called, can take 7 seconds!
SOLUTION:
I've created a patch that introduces an option "optimize-frequency" to determine the frequency of the optimize call.
It defaults to 1 (current behaviour), when a user sets it to 50, only once every 50 updates the index will be optimized etc....
If no optimization is wanted, you can set it to 0.
This is compliant to the Lucene documentation (fragment of Lucene FAQ):
"The IndexWriter class supports an optimize() method that compacts the index database and speedup queries. You may want to use this method after performing a complete indexing of your document set or after incremental updates of the index. If your incremental update adds documents frequently, you want to perform the optimization only once in a while to avoid the extra overhead of the optimization."
PATCH INFO:
added configuration option + a function "needToOptimize()" which is called before optimizing.
needToOptimize() uses a random function generator, to keep code simple.
- when the option is not set, CODE WILL BE EXECUTED AS BEFORE
- tested one 2.1.11 SVN branch, but no differences in the "main" trunk thus can be applied there also.
- Updated API docs
- if patch accepted, I will also update the Wiki:
http://wiki.apache.org/cocoon/LuceneIndexTransformer
The LuceneIndexTransformer optimizes the Lucene index every time you add an entry to the index.
This slows down enormously the indexing with a large index ! If upon every checkin of a document eg,
you use it to update the entry, it will slow down.
Eg. I have a Pentium IV 2.4 Ghz, Lucene index contains 10 000 doc.
Where the index update only takes say 60ms, the optimize that get's called, can take 7 seconds!
SOLUTION:
I've created a patch that introduces an option "optimize-frequency" to determine the frequency of the optimize call.
It defaults to 1 (current behaviour), when a user sets it to 50, only once every 50 updates the index will be optimized etc....
If no optimization is wanted, you can set it to 0.
This is compliant to the Lucene documentation (fragment of Lucene FAQ):
"The IndexWriter class supports an optimize() method that compacts the index database and speedup queries. You may want to use this method after performing a complete indexing of your document set or after incremental updates of the index. If your incremental update adds documents frequently, you want to perform the optimization only once in a while to avoid the extra overhead of the optimization."
PATCH INFO:
added configuration option + a function "needToOptimize()" which is called before optimizing.
needToOptimize() uses a random function generator, to keep code simple.
- when the option is not set, CODE WILL BE EXECUTED AS BEFORE
- tested one 2.1.11 SVN branch, but no differences in the "main" trunk thus can be applied there also.
- Updated API docs
- if patch accepted, I will also update the Wiki:
http://wiki.apache.org/cocoon/LuceneIndexTransformer