Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5591

ReaderAndUpdates should create a proper IOContext when writing DV updates

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Today we pass IOContext.DEFAULT. If DV updates are used in conjunction w/ NRTCachingDirectory, it means the latter will attempt to write the entire DV field in its RAMDirectory, which could lead to OOM.

      Would be good if we can build our own FlushInfo, estimating the number of bytes we're about to write. I didn't see off hand a quick way to guesstimate that - I thought to use the current DV's sizeInBytes as an approximation, but I don't see a way to get it, not a direct way at least.

      Maybe we can use the size of the in-memory updates to guesstimate that amount? Something like sizeOfInMemUpdates * (maxDoc/numUpdatedDocs)? Is it a too wild guess?

        Attachments

        1. LUCENE-5591.patch
          10 kB
          Shai Erera
        2. LUCENE-5591.patch
          14 kB
          Shai Erera
        3. LUCENE-5591.patch
          14 kB
          Shai Erera

          Activity

            People

            • Assignee:
              shaie Shai Erera
              Reporter:
              shaie Shai Erera
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: