Lucene - Core
  1. Lucene - Core
  2. LUCENE-2357

Reduce transient RAM usage while merging by using packed ints array for docID re-mapping

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA, Trunk
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We allocate this int[] to remap docIDs due to compaction of deleted ones.

      This uses alot of RAM for large segment merges, and can fail to allocate due to fragmentation on 32 bit JREs.

      Now that we have packed ints, a simple fix would be to use a packed int array... and maybe instead of storing abs docID in the mapping, we could store the number of del docs seen so far (so the remap would do a lookup then a subtract). This may add some CPU cost to merging but should bring down transient RAM usage quite a bit.

      1. LUCENE-2357.patch
        17 kB
        Adrien Grand
      2. LUCENE-2357.patch
        17 kB
        Michael McCandless
      3. LUCENE-2357.patch
        16 kB
        Adrien Grand
      4. LUCENE-2357.patch
        16 kB
        Michael McCandless
      5. LUCENE-2357.patch
        15 kB
        Adrien Grand

        Issue Links

          Activity

          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Adrien Grand made changes -
          Link This issue relates to LUCENE-4792 [ LUCENE-4792 ]
          Adrien Grand made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Assignee Michael McCandless [ mikemccand ] Adrien Grand [ jpountz ]
          Resolution Fixed [ 1 ]
          Michael McCandless made changes -
          Fix Version/s 5.0 [ 12321663 ]
          Michael McCandless made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Michael McCandless made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 4.0 [ 12314025 ]
          Fix Version/s 4.1 [ 12321140 ]
          Resolution Fixed [ 1 ]
          Adrien Grand made changes -
          Attachment LUCENE-2357.patch [ 12530057 ]
          Michael McCandless made changes -
          Attachment LUCENE-2357.patch [ 12530029 ]
          Adrien Grand made changes -
          Attachment LUCENE-2357.patch [ 12529991 ]
          Michael McCandless made changes -
          Attachment LUCENE-2357.patch [ 12529978 ]
          Michael McCandless made changes -
          Labels gsoc2012 lucene-gsoc-12
          Assignee Michael McCandless [ mikemccand ]
          Adrien Grand made changes -
          Attachment LUCENE-2357.patch [ 12529913 ]
          Robert Muir made changes -
          Fix Version/s 4.1 [ 12321140 ]
          Fix Version/s 4.0 [ 12314025 ]
          Michael McCandless made changes -
          Labels gsoc2012 lucene-gsoc-12
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12563588 ] jira [ 12585173 ]
          Mark Thomas made changes -
          Field Original Value New Value
          Workflow jira [ 12503243 ] Default workflow, editable Closed status [ 12563588 ]
          Michael McCandless created issue -

            People

            • Assignee:
              Adrien Grand
              Reporter:
              Michael McCandless
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development