[SOLR-14645] Make FileListEntityProcessor streaming to prevent OOM

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: contrib - DataImportHandler
• Assignee: Unassigned
• Reporter: Marco Bolis
• Created: 13/Jul/20 10:28
• Updated: 21/Oct/20 11:08
• Resolved: 29/Aug/20 20:34

Description

FileListEntityProcessor currently works by keeping the whole list of files to process in memory and iterating over it.
If the list is very large (think some 100s of Ks) Solr goes OOM and crashes.
Using Java 8 Streams and NIO2 it is possible to entirely avoid this behavior.

Issue Links

• is superceded by: SOLR-14783 Remove DIH from 9.0
• links to: GitHub Pull Request #1667