Lucene - Core
  1. Lucene - Core
  2. LUCENE-1168

TermVectors index files can become corrupt when autoCommit=false

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3
    • Fix Version/s: 2.3.1, 2.4
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff from this thread:

      http://www.gossamer-threads.com/lists/lucene/java-dev/55951

      There are actually 2 separate cases here, both only happening when
      autoCommit=false:

      • First issue was caused by LUCENE-843 (sigh): if you add a bunch of
        docs with no term vectors, such that 1 or more flushes happen;
        then you add docs that do have term vectors, the tvx file will not
        have enough entries (= corruption).
      • Second issue was caused by bulk merging of term vectors
        (LUCENE-1120 – only in trunk) and bulk merging of stored fields
        (LUCENE-1043, in 2.3), and only shows when autoCommit=false, and,
        the bulk merging optimization runs. In this case, the code that
        reads the rawDocs tries to read too far in the tvx/fdx files (it's
        not really index corruption but rather a bug in the rawDocs
        reading).
      1. LUCENE-1168.patch
        8 kB
        Michael McCandless

        Issue Links

          Activity

          Hide
          Michael McCandless added a comment -

          Patch attached. I added the original test case as a unit test, plus 2
          other tests.

          The fixes themselves are trivial (a few lines). I plan to commit end
          of today.

          Thanks for finding & reporting this Suresh!

          Show
          Michael McCandless added a comment - Patch attached. I added the original test case as a unit test, plus 2 other tests. The fixes themselves are trivial (a few lines). I plan to commit end of today. Thanks for finding & reporting this Suresh!
          Hide
          Michael McCandless added a comment -

          Backported to 2.3 branch.

          Show
          Michael McCandless added a comment - Backported to 2.3 branch.

            People

            • Assignee:
              Michael McCandless
              Reporter:
              Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development