Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14645

Make FileListEntityProcessor streaming to prevent OOM

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      FileListEntityProcessor currently works by keeping the whole list of files to process in memory and iterating over it.
      If the list is very large (think some 100s of Ks) Solr goes OOM and crashes.
      Using Java 8 Streams and NIO2 it is possible to entirely avoid this behavior.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mbolis Marco Bolis
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m