Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2467

IndexWriter memory leak when large docs are indexed

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3, 2.3.1, 2.3.2, 2.4, 2.4.1, 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2, 3.1, 4.0-ALPHA
    • 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
    • core/index
    • None
    • New

    Description

      Spinoff from the java-user thread "IndexWriter and memory usage"...

      IndexWriter has had a long standing memory leak, since LUCENE-843.

      When the byte/char/int blocks are recycled to the common pool, the
      per-thread DW classes incorrectly still hold a reference to them.

      This normally is not a problem, since these buffers will be re-used
      again.

      But, if you index a massive document, causing IW to allocate more than
      the RAM buffer allocated to it, then the leak happens. So you could
      have a 16 MB RAM buffer set, but if a huge doc required allocation of
      200 MB worth of arrays, those 200 MB are never freed (well, until you
      close the IW and deref it from the app).

      It's even worse if you use multiple threads: if each thread has ever
      had to index a massive document, then that thread incorrectly holds
      onto the extra arrays.

      Attachments

        1. LUCENE-2467.patch
          3 kB
          Michael McCandless
        2. LUCENE-2467.patch
          3 kB
          Michael McCandless

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mikemccand Michael McCandless
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment