1. Cocoon
  2. COCOON-2065

huge performance increase of LuceneIndexTransformer on large Lucene indexes


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.1.11, 2.2
    • Fix Version/s: 2.1.12, 2.2
    • Component/s: Blocks: Lucene
    • Labels:
    • Urgency:
    • Other Info:
      Patch available



      The LuceneIndexTransformer optimizes the Lucene index every time you add an entry to the index.
      This slows down enormously the indexing with a large index ! If upon every checkin of a document eg,
      you use it to update the entry, it will slow down.

      Eg. I have a Pentium IV 2.4 Ghz, Lucene index contains 10 000 doc.
      Where the index update only takes say 60ms, the optimize that get's called, can take 7 seconds!


      I've created a patch that introduces an option "optimize-frequency" to determine the frequency of the optimize call.
      It defaults to 1 (current behaviour), when a user sets it to 50, only once every 50 updates the index will be optimized etc....
      If no optimization is wanted, you can set it to 0.

      This is compliant to the Lucene documentation (fragment of Lucene FAQ):

      "The IndexWriter class supports an optimize() method that compacts the index database and speedup queries. You may want to use this method after performing a complete indexing of your document set or after incremental updates of the index. If your incremental update adds documents frequently, you want to perform the optimization only once in a while to avoid the extra overhead of the optimization."


      added configuration option + a function "needToOptimize()" which is called before optimizing.
      needToOptimize() uses a random function generator, to keep code simple.

      - when the option is not set, CODE WILL BE EXECUTED AS BEFORE
      - tested one 2.1.11 SVN branch, but no differences in the "main" trunk thus can be applied there also.
      - Updated API docs
      - if patch accepted, I will also update the Wiki:


        Dominique De Munck created issue -
        Dominique De Munck made changes -
        Field Original Value New Value
        Attachment LuceneIndexTransformer.patch [ 12357464 ]
        Felix Knecht made changes -
        Assignee Felix Knecht [ felixk ]
        Felix Knecht made changes -
        Fix Version/s 2.2-dev (Current SVN) [ 12310611 ]
        Felix Knecht made changes -
        Assignee Felix Knecht [ felixk ]
        Felix Knecht made changes -
        Assignee Felix Knecht [ felixk ]
        Felix Knecht made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Antonio Gallardo made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Felix Knecht made changes -
        Fix Version/s 2.1.11-dev (Current SVN) [ 12312231 ]
        Assignee Felix Knecht [ felixk ]
        Alfred Nathaniel made changes -
        Assignee Alfred Nathaniel [ ]
        Alfred Nathaniel made changes -
        Fix Version/s 2.1.12-dev (Current SVN) [ 12312903 ]
        Alfred Nathaniel made changes -
        Resolution Fixed [ 1 ]
        Status Reopened [ 4 ] Closed [ 6 ]


          • Assignee:
            Alfred Nathaniel
            Dominique De Munck
          • Votes:
            0 Vote for this issue
            0 Start watching this issue


            • Created: