Lucene - Core
  1. Lucene - Core
  2. LUCENE-5263

Deletes may be silently lost if an IOException is hit and later not hit (e.g., disk fills up and then frees up)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.5.1, 4.6, Trunk
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This case is tricky to handle, yet I think realistic: disk fills up
      temporarily, causes an exception in writeLiveDocs, and then the app
      keeps using the IW instance.

      Meanwhile disk later frees up again, IW is closed "successfully". In
      certain cases, we can silently lose deletes in this case.

      I had already committed
      TestIndexWriterDeletes.testNoLostDeletesOnDiskFull, and Jenkins seems
      happy with it so far, but when I added fangs to the test (cutover to
      RandomIndexWriter from IndexWriter, allow IOE during getReader, add
      randomness to when exc is thrown, etc.), it uncovered some real/nasty
      bugs:

      • ReaderPool.dropAll was suppressing any exception it hit, because
        if (priorE != null)

        should instead be

        if (priorE == null)
      • After a merge, we have to write deletes before committing the
        segment, because an exception when writing deletes means we need
        to abort the merge
      • Several places that were directly calling deleter.checkpoint must
        also increment the changeCount else on close IW thinks there are
        no changes and doesn't write a new segments file.
      • closeInternal was dropping pooled readers after writing the
        segments file, which would lose deletes still buffered due to a
        previous exc.
      1. LUCENE-5263.patch
        16 kB
        Michael McCandless
      2. LUCENE-5263.patch
        33 kB
        Michael McCandless

        Activity

        Michael McCandless created issue -
        Michael McCandless made changes -
        Field Original Value New Value
        Attachment LUCENE-5263.patch [ 12607261 ]
        Michael McCandless made changes -
        Attachment LUCENE-5263.patch [ 12607351 ]
        Michael McCandless made changes -
        Summary Deletes may be silently lost if disk fills up and then frees up Deletes may be silently lost if an IOException is hit and later not hit (e.g., disk fills up and then frees up)
        Michael McCandless made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Robert Muir made changes -
        Fix Version/s 4.5.1 [ 12325265 ]

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development