[SOLR-3443] Optimize hunspell dictionary loading with multiple cores

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Luca Cavanna
• Created: 07/May/12 12:35
• Updated: 14/May/24 12:48

Description

The Hunspell dictionary is actually loaded into memory. Each core using hunspell loads its own dictionary, no matter if all the cores are using the same dictionary files. As a result, the same dictionary is loaded into memory multiple times, once for each core. I think we should share those dictionaries between all cores in order to optimize the memory usage. In fact, let's say a dictionary takes 20MB into memory (this is what I detected), if you have 20 cores you are going to use 400MB only for dictionaries, which doesn't seem a good idea to me.