This issue was first reported in LUCENENET-640.
When using very high number of threads for long periods of time, the garbage collector will cause all threads to block for several seconds. More details here.
There is also a solution being tested, which is blogged about here.
Ideally, we should adapt the fix after it has been thoroughly tested, provided Microsoft doesn't fix the problem with ThreadLocal<T> first.