Lucene - Core
  1. Lucene - Core
  2. LUCENE-1168

TermVectors index files can become corrupt when autoCommit=false

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3
    • Fix Version/s: 2.3.1, 2.4
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff from this thread:

      http://www.gossamer-threads.com/lists/lucene/java-dev/55951

      There are actually 2 separate cases here, both only happening when
      autoCommit=false:

      • First issue was caused by LUCENE-843 (sigh): if you add a bunch of
        docs with no term vectors, such that 1 or more flushes happen;
        then you add docs that do have term vectors, the tvx file will not
        have enough entries (= corruption).
      • Second issue was caused by bulk merging of term vectors
        (LUCENE-1120 – only in trunk) and bulk merging of stored fields
        (LUCENE-1043, in 2.3), and only shows when autoCommit=false, and,
        the bulk merging optimization runs. In this case, the code that
        reads the rawDocs tries to read too far in the tvx/fdx files (it's
        not really index corruption but rather a bug in the rawDocs
        reading).
      1. LUCENE-1168.patch
        8 kB
        Michael McCandless

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          6h 22m 1 Michael McCandless 07/Feb/08 21:15
          Resolved Resolved Closed Closed
          16d 3h 24m 1 Michael Busch 24/Feb/08 00:40
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12564702 ] jira [ 12585039 ]
          Mark Thomas made changes -
          Workflow jira [ 12422991 ] Default workflow, editable Closed status [ 12564702 ]
          Michael Busch made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hoss Man made changes -
          Fix Version/s 2.3.1 [ 12312975 ]
          Grant Ingersoll made changes -
          Link This issue blocks SOLR-342 [ SOLR-342 ]
          Hide
          Michael McCandless added a comment -

          Backported to 2.3 branch.

          Show
          Michael McCandless added a comment - Backported to 2.3 branch.
          Michael McCandless made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Michael McCandless made changes -
          Field Original Value New Value
          Attachment LUCENE-1168.patch [ 12374975 ]
          Hide
          Michael McCandless added a comment -

          Patch attached. I added the original test case as a unit test, plus 2
          other tests.

          The fixes themselves are trivial (a few lines). I plan to commit end
          of today.

          Thanks for finding & reporting this Suresh!

          Show
          Michael McCandless added a comment - Patch attached. I added the original test case as a unit test, plus 2 other tests. The fixes themselves are trivial (a few lines). I plan to commit end of today. Thanks for finding & reporting this Suresh!
          Michael McCandless created issue -

            People

            • Assignee:
              Michael McCandless
              Reporter:
              Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development