Lucene - Core
  1. Lucene - Core
  2. LUCENE-1566

Large Lucene index can hit false OOM due to Sun JRE issue

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 2.9
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is not a Lucene issue, but I want to open this so future google
      diggers can more easily find it.

      There's this nasty bug in Sun's JRE:

      http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6478546

      The gist seems to be, if you try to read a large (eg 200 MB) number of
      bytes during a single RandomAccessFile.read call, you can incorrectly
      hit OOM. Lucene does this, with norms, since we read in one byte per
      doc per field with norms, as a contiguous array of length maxDoc().

      The workaround was a custom patch to do large file reads as several
      smaller reads.

      Background here:

      http://www.nabble.com/problems-with-large-Lucene-index-td22347854.html

      1. LUCENE-1566.patch
        4 kB
        Simon Willnauer
      2. LUCENE-1566.patch
        4 kB
        Simon Willnauer
      3. LUCENE-1566.patch
        18 kB
        Michael McCandless
      4. LUCENE_1566_IndexInput.patch
        17 kB
        Simon Willnauer
      5. LUCENE_1566_IndexInput_Changes.patch
        14 kB
        Simon Willnauer
      6. LUCENE_1566_IndexInput_Changes.patch
        21 kB
        Simon Willnauer

        Activity

        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563149 ] jira [ 12584231 ]
        Mark Thomas made changes -
        Workflow jira [ 12454946 ] Default workflow, editable Closed status [ 12563149 ]
        Mark Miller made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Michael McCandless made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Michael McCandless made changes -
        Attachment LUCENE-1566.patch [ 12413699 ]
        Simon Willnauer made changes -
        Attachment LUCENE_1566_IndexInput_Changes.patch [ 12413671 ]
        Simon Willnauer made changes -
        Attachment LUCENE_1566_IndexInput_Changes.patch [ 12413664 ]
        Simon Willnauer made changes -
        Attachment LUCENE_1566_IndexInput.patch [ 12413570 ]
        Michael McCandless made changes -
        Assignee Michael McCandless [ mikemccand ] Simon Willnauer [ simonw ]
        Michael McCandless made changes -
        Fix Version/s 2.9 [ 12312682 ]
        Michael McCandless made changes -
        Assignee Simon Willnauer [ simonw ] Michael McCandless [ mikemccand ]
        Simon Willnauer made changes -
        Attachment LUCENE-1566.patch [ 12412258 ]
        Simon Willnauer made changes -
        Attachment LUCENE-1566.patch [ 12412087 ]
        Simon Willnauer made changes -
        Field Original Value New Value
        Assignee Simon Willnauer [ simonw ]
        Michael McCandless created issue -

          People

          • Assignee:
            Simon Willnauer
            Reporter:
            Michael McCandless
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development