Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7894

IndexWriter suffers temporary short-term amnesia

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Nightly benchmarks had been failing to run for a while due to trunk API changes, but when I finally fixed those, the indexer fails because IW.maxDoc disagrees with the number of documents indexed after all threads are done indexing.

      It's sort of crazy none of our tests caught this!

      I tracked down the bug; it was caused in LUCENE-7868. The bug is not as bad as it sounds: documents are not in fact lost, it's just IW's internal accounting for maxDoc that's temporarily incorrect until you do a refresh or commit or close the index.

      I'm marking as blocker for 7.0.

      1. LUCENE-7894.patch
        6 kB
        Michael McCandless

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ee1edd9d467bbccfe7015ec5ebee93f8b5bd4b14 in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee1edd9 ]

        LUCENE-7894: make sure IW finishes segment flush so we don't see a temporary drop in IW.maxDoc()

        Show
        jira-bot ASF subversion and git services added a comment - Commit ee1edd9d467bbccfe7015ec5ebee93f8b5bd4b14 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee1edd9 ] LUCENE-7894 : make sure IW finishes segment flush so we don't see a temporary drop in IW.maxDoc()
        Hide
        mikemccand Michael McCandless added a comment -

        Simple patch w/ test case showing the issue. The problem was IW was failing to finish flushing a segment with the indexing thread that just wrote all the files for the segment.

        Show
        mikemccand Michael McCandless added a comment - Simple patch w/ test case showing the issue. The problem was IW was failing to finish flushing a segment with the indexing thread that just wrote all the files for the segment.

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development