Lucene - Core
  1. Lucene - Core
  2. LUCENE-4629

IndexWriter fails to delete documents if Iterator<IndexDocument> throws an exception

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0
    • Fix Version/s: 4.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      In DWPT we iterator over a document block and roll back documents if one of the docs fails with a non-aborting exception. Yet, we miss to delete those document if the iterator itself throws an exception. Given the fact that we allow an Iterable on IW we should be prepared for RT exceptions since these documents might be created in a stream fashing rather than already build up. IMO its a valid usecase if you have large documents to not materialize them in memory before indexing or at least we don't require this.

      1. LUCENE-4629.patch
        4 kB
        Simon Willnauer

        Activity

        Hide
        Simon Willnauer added a comment -

        here is a testcase and a fix

        Show
        Simon Willnauer added a comment - here is a testcase and a fix
        Hide
        Michael McCandless added a comment -

        Nice catch! Sneaky.

        +1 to remove entire block if anything goes wrong, and +1 for patch.

        Thanks Simon.

        Show
        Michael McCandless added a comment - Nice catch! Sneaky. +1 to remove entire block if anything goes wrong, and +1 for patch. Thanks Simon.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Simon Willnauer
        http://svn.apache.org/viewvc?view=revision&revision=1421801

        LUCENE-4629: IndexWriter fails to delete documents if Iterator<IndexDocument> throws an exception

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Simon Willnauer http://svn.apache.org/viewvc?view=revision&revision=1421801 LUCENE-4629 : IndexWriter fails to delete documents if Iterator<IndexDocument> throws an exception
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Simon Willnauer
        http://svn.apache.org/viewvc?view=revision&revision=1421798

        LUCENE-4629: IndexWriter fails to delete documents if Iterator<IndexDocument> throws an exception

        Show
        Commit Tag Bot added a comment - [trunk commit] Simon Willnauer http://svn.apache.org/viewvc?view=revision&revision=1421798 LUCENE-4629 : IndexWriter fails to delete documents if Iterator<IndexDocument> throws an exception

          People

          • Assignee:
            Unassigned
            Reporter:
            Simon Willnauer
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development