Lucene - Core
  1. Lucene - Core
  2. LUCENE-2759

We should never open an IndexInput when an IndexOutput is still open

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.4, 3.0.3, 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I modified MockDirWrapper to assert this (except for
      segments_N/segments.gen, where it's expected), and, it uncovered a
      couple of places involving NRT readers where we open a shared doc
      store file that's still open for writing.

      First, if you install a merged segment warmer, we were failing to
      force the merge of the doc stores in this case, thus potentially
      opening the same doc stores that are also still open for writing.

      Second, if you're actively adding docs in other threads when you call
      IW.getReader(), the other threads could sneak in and flush new
      segments sharing the doc stores. The returned reader then opens the
      doc store files that are still open for writing.

      1. LUCENE-2759.patch
        23 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch.

        I also fixed a concurrency issue with MockDirWrapper, and an intermittent failure in TestCachingWrapper/SpanFilter.

        Show
        Michael McCandless added a comment - Patch. I also fixed a concurrency issue with MockDirWrapper, and an intermittent failure in TestCachingWrapper/SpanFilter.
        Hide
        Jason Rutherglen added a comment -

        RT's usage of doc stores, ie, reading from them while they're still being written to will likely break this.

        Show
        Jason Rutherglen added a comment - RT's usage of doc stores, ie, reading from them while they're still being written to will likely break this.
        Hide
        Michael McCandless added a comment -

        RT's usage of doc stores, ie, reading from them while they're still being written to will likely break this.

        True, we need to cross that bridge w/ RT.

        Actually I think Michael & I discussed it already on an issue but I can't remember where.

        Show
        Michael McCandless added a comment - RT's usage of doc stores, ie, reading from them while they're still being written to will likely break this. True, we need to cross that bridge w/ RT. Actually I think Michael & I discussed it already on an issue but I can't remember where.
        Hide
        Jason Rutherglen added a comment -

        Actually I think Michael & I discussed it already on an issue but I can't remember where.

        Yeah we discussed it before in LUCENE-1313, and possibly in the RT issue. In the LUCENE-2312 patch the doc stores are being flushed on get reader and the doc stores read successfully.

        Show
        Jason Rutherglen added a comment - Actually I think Michael & I discussed it already on an issue but I can't remember where. Yeah we discussed it before in LUCENE-1313 , and possibly in the RT issue. In the LUCENE-2312 patch the doc stores are being flushed on get reader and the doc stores read successfully.
        Hide
        Michael McCandless added a comment -

        In the LUCENE-2312 patch the doc stores are being flushed on get reader and the doc stores read successfully.

        Flushed by not closed? And that works fine on Windows? That's fascinating to me

        Ie, that our dir impls are able to get an IndexInput when there's still an IndexOutput writing to it.

        But for RT this will be necessary. I had thought that we'd have to extend Directory to explicitly add eg an IndexInputOutput, ie designed for full read/write.

        Show
        Michael McCandless added a comment - In the LUCENE-2312 patch the doc stores are being flushed on get reader and the doc stores read successfully. Flushed by not closed? And that works fine on Windows? That's fascinating to me Ie, that our dir impls are able to get an IndexInput when there's still an IndexOutput writing to it. But for RT this will be necessary. I had thought that we'd have to extend Directory to explicitly add eg an IndexInputOutput, ie designed for full read/write.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development