Lucene - Core
  1. Lucene - Core
  2. LUCENE-1125

Excessive Arrays.fill(0) in DocumentsWriter drastically slows down small docs (3.9X slowdown!)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3
    • Fix Version/s: 2.3
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I've been doing some "final" performance testing of 2.3RC1 and
      uncovered a fairly serious bug that adds a large fixed CPU cost when
      documents have any term vector enabled fields.

      The bug does not affect correctness, just performance.

      Basically, for every document, we were calling Arrays.fill(0) on a
      large (32 KB) byte array when in fact we only needed to zero a small
      part of it. This only happens if term vectors are turned on, and is
      especially devastating for small documents.

      1. LUCENE-1125.patch
        2 kB
        Michael McCandless

        Activity

        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12562460 ] jira [ 12584767 ]
        Mark Thomas made changes -
        Workflow jira [ 12420613 ] Default workflow, editable Closed status [ 12562460 ]
        Michael Busch made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Michael McCandless made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Michael McCandless added a comment -

        Fixed on trunk & 2.3 branch.

        Show
        Michael McCandless added a comment - Fixed on trunk & 2.3 branch.
        Hide
        Michael McCandless added a comment -

        OK, why don't you commit this today. Meanwhile I'll look into the
        small issues Hoss pointed out and then build RC2 end of today.

        OK, will do, thanks!

        Show
        Michael McCandless added a comment - OK, why don't you commit this today. Meanwhile I'll look into the small issues Hoss pointed out and then build RC2 end of today. OK, will do, thanks!
        Hide
        Michael Busch added a comment -

        The fix is very low risk. All tests pass.

        Yes, all contrib & core tests pass for me too. And after reading the
        patch I agree that it looks good and is low risk.

        Michael, I think we should spin 2.3 RC2 to include this fix? Sorry to
        only find it so late in the game

        OK, why don't you commit this today. Meanwhile I'll look into the
        small issues Hoss pointed out and then build RC2 end of today.

        Oh, and no need to be sorry! This is what the code freeze period is
        for - finding and fixing problems!

        Show
        Michael Busch added a comment - The fix is very low risk. All tests pass. Yes, all contrib & core tests pass for me too. And after reading the patch I agree that it looks good and is low risk. Michael, I think we should spin 2.3 RC2 to include this fix? Sorry to only find it so late in the game OK, why don't you commit this today. Meanwhile I'll look into the small issues Hoss pointed out and then build RC2 end of today. Oh, and no need to be sorry! This is what the code freeze period is for - finding and fixing problems!
        Michael McCandless made changes -
        Field Original Value New Value
        Attachment LUCENE-1125.patch [ 12372905 ]
        Hide
        Michael McCandless added a comment -

        Attached patch. I ran a test where I index the first 6M docs from
        Wikipedia preprocessed to 100 bytes each, using this alg:

          analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer
          doc.maker=org.apache.lucene.benchmark.byTask.feeds.LineDocMaker
          work.dir=/lucene/work
          doc.stored = true
          doc.term.vector = true
          doc.term.vector.positions = true
          doc.term.vector.offsets = true
          ram.flush.mb = 64
          compound = false
          autocommit = false
          docs.file=/Volumes/External/lucene/wikifull100.txt
          doc.add.log.step=10000
          
          directory=FSDirectory
          
          ResetSystemErase
          { "BuildIndex"
            CreateIndex
            [ { "AddDocs" AddDoc > : 1500000 ]: 4
            CloseIndex
          }
          
          RepSumByPrefRound BuildIndex
        

        With this fix, it takes 158.5 seconds. Without it, it takes 621.8
        seconds = 3.9X slower!

        The fix is very low risk. All tests pass.

        Michael, I think we should spin 2.3 RC2 to include this fix? Sorry to
        only find it so late in the game

        Show
        Michael McCandless added a comment - Attached patch. I ran a test where I index the first 6M docs from Wikipedia preprocessed to 100 bytes each, using this alg: analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer doc.maker=org.apache.lucene.benchmark.byTask.feeds.LineDocMaker work.dir=/lucene/work doc.stored = true doc.term.vector = true doc.term.vector.positions = true doc.term.vector.offsets = true ram.flush.mb = 64 compound = false autocommit = false docs.file=/Volumes/External/lucene/wikifull100.txt doc.add.log.step=10000 directory=FSDirectory ResetSystemErase { "BuildIndex" CreateIndex [ { "AddDocs" AddDoc > : 1500000 ]: 4 CloseIndex } RepSumByPrefRound BuildIndex With this fix, it takes 158.5 seconds. Without it, it takes 621.8 seconds = 3.9X slower! The fix is very low risk. All tests pass. Michael, I think we should spin 2.3 RC2 to include this fix? Sorry to only find it so late in the game
        Michael McCandless created issue -

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development