Lucene - Core
  1. Lucene - Core
  2. LUCENE-2097

In NRT mode, and CFS enabled, IndexWriter incorrectly ties up disk space

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.9, 2.9.1, 3.0
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff of java-user thread titled "searching while optimize"...

      If IndexWriter is in NRT mode (you've called getReader() at least
      once), and CFS is enabled, then internally the writer pools readers.
      However, after a merge completes, it opens the reader against het
      non-CFS segment files, and pools that. It then builds the CFS file,
      as well, thus tying up the storage for that segment twice.

      Functionally the bug is harmless (it's only a disk space issue).
      Also, when the segment is merged, the disk space is released again
      (though the newly merged segment will also be double-tied-up).

      Simple workaround is to use non-CFS mode, or, don't use getReader.

      1. LUCENE-2097.patch
        2 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Attached patch with test case that shows the issue. Not yet sure what's the best way to fix it... probably we have to build the CFS before opening the reader we want to pool.

        Show
        Michael McCandless added a comment - Attached patch with test case that shows the issue. Not yet sure what's the best way to fix it... probably we have to build the CFS before opening the reader we want to pool.
        Hide
        Michael McCandless added a comment -

        This was separately fixed – we now build the CFS before cutting over to it in the in-memory segments.

        Show
        Michael McCandless added a comment - This was separately fixed – we now build the CFS before cutting over to it in the in-memory segments.
        Hide
        Grant Ingersoll added a comment -

        Bulk close for 3.1

        Show
        Grant Ingersoll added a comment - Bulk close for 3.1

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development