[LUCENENET-640] Sequential IndexWriter performance in concurrent environments.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Lucene.Net 4.8.0
• Fix Version/s: Lucene.Net 4.8.0
• Component/s: Lucene.Net Core
• Labels: performance
• Flags: Important
• Assignee: Unassigned
• Reporter: Mathias Henriksen
• Created: 13/Jan/20 16:35
• Updated: 09/Dec/21 17:49
• Resolved: 09/Dec/21 17:48

Description

When creating Lucene.Net indices in parallel, sequential-like performance is experienced. Profiling 8 concurrent IndexWriter instances writing in parallel shows that WeakIdentityMap::IdentityWeakReference::Equals spends most time garbage collecting (94.91%) and TokenStream::AssertFinal (87.09% garbage collecting) in my preliminary tests (see screenshots).

The WeakIdentityMap implementation uses an IdentityWeakReference as key, which is implemented as a class. By inspection of this class, it is merely a System.Runtime.InteropServices.GCHandle wrapper as can be seen in the mono project, manually wrapping of this struct in a struct rather than a class - will eliminate some of the immense amounts of garbage collection.

Issue Links

• Is contained by: LUCENENET-630 Identify/Fix Bottlenecks
• is related to: LUCENENET-616 Make Collections from Lucene.Net.Support into a 1st Class Feature
• is related to: LUCENENET-610 Reduce locking in FieldCacheImpl::Cache::Get
• is related to: LUCENENET-636 Port ConditionalWeakTable from .NET Core 3.x to .NET Standard 2.0
• relates to: LUCENENET-644 The design and implementation of a better ThreadLocal<T>
• relates to: LUCENENET-642 Eliminate Runtime Reflection Code from TokenStream
• links to: GitHub Pull Request #236