Lucene - Core
  1. Lucene - Core
  2. LUCENE-3627

CorruptIndexException on indexing after a failure occurs after segments file creation but before any bytes are written

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Lucene Fields:
      New, Patch Available

      Description

      FSDirectory.createOutput(..) uses a RandomAccessFile to do its work. On my system the default FSDirectory.open(..) creates an NIOFSDirectory. If createOutput is called on a segments_* file and a crash occurs between RandomAccessFile creation (file system shows a segments_* file exists but has zero bytes) but before any bytes are written to the file, subsequent IndexWriters cannot proceed. The difficulty is that it does not know how to clear the empty segments_* file. None of the file deletions will happen on such a segment file because the opening bytes cannot not be read to determine format and version.

      An initial proposed patch file is attached below.

      1. LUCENE-3627_initial_proposal.txt
        14 kB
        Ken McCracken
      2. LUCENE-3627.patch
        10 kB
        Michael McCandless
      3. TestCrashCausesCorruptIndex.java
        11 kB
        Ken McCracken

        Activity

        Ken McCracken created issue -
        Ken McCracken made changes -
        Field Original Value New Value
        Attachment TestCrashCausesCorruptIndex.java [ 12506505 ]
        Ken McCracken made changes -
        Affects Version/s 3.5 [ 12317877 ]
        Environment Mac OS X 10.6.5, running tests in Eclipse Build id: 20100218-1602,
        java version "1.6.0_24"
        Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
        Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
        lucene-3.5.0, src download from GA release lucene.apache.org.
        Mac OS X 10.6.5, running tests in Eclipse Build id: 20100218-1602,
        java version "1.6.0_24"
        Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
        Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
        Ken McCracken made changes -
        Attachment LUCENE-3627_initial_proposal.txt [ 12506539 ]
        Ken McCracken made changes -
        Lucene Fields New [ 10121 ] New,Patch Available [ 10121,10120 ]
        Description FSDirectory.createOutput(..) uses a RandomAccessFile to do its work. On my system the default FSDirectory.open(..) creates an NIOFSDirectory. If createOutput is called on a segments_* file and a crash occurs between RandomAccessFile creation (file system shows a segments_* file exists but has zero bytes) but before any bytes are written to the file, subsequent IndexWriters cannot proceed. The difficulty is that it does not know how to clear the empty segments_* file. None of the file deletions will happen on such a segment file because the opening bytes cannot not be read to determine format and version.

        I will attempt to attach a Test file demonstrates the issue; place it in your
        src/test/org/apache/lucene/store/
        directory and run the unit tests with JUnit4.

        FSDirectory.createOutput(..) uses a RandomAccessFile to do its work. On my system the default FSDirectory.open(..) creates an NIOFSDirectory. If createOutput is called on a segments_* file and a crash occurs between RandomAccessFile creation (file system shows a segments_* file exists but has zero bytes) but before any bytes are written to the file, subsequent IndexWriters cannot proceed. The difficulty is that it does not know how to clear the empty segments_* file. None of the file deletions will happen on such a segment file because the opening bytes cannot not be read to determine format and version.

        An initial proposed patch file is attached below.

        Michael McCandless made changes -
        Assignee Michael McCandless [ mikemccand ]
        Michael McCandless made changes -
        Attachment LUCENE-3627.patch [ 12506600 ]
        Michael McCandless made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 3.6 [ 12319070 ]
        Fix Version/s 4.0 [ 12314025 ]
        Resolution Fixed [ 1 ]
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Ken McCracken
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development