Just to clear up on some comments:
Guys I'm testing this on a production deploy and the use case is simple. Parser Query, Search, Get Doc fields. Contention was seen without profiling and profiling was used to find where the contention in the application.
|I don't care. They should be reusing tokenstreams in their application.
Token streams is not the only place of contention discussed here. Please look at the test attached (App.java). It does not recreate the token streams on each search, only (by mistake) for each thread. But still this only has a very small impact. Searches are repeated from each thread reusing the same token stream.
|We don't need to make lucene more complicated to (possibly) speed up someones broken code.
Please write an example and show me your findings. I've done the same already. Have a look at App.java
|I want to see the QPS change, I don't trust the benchmarks or the various fancy jvm tools being used on this issue.
I think its all ghosts.
Yourkit is a standard Profiling tool. I didn't just wake up and say hey I'll profile lucene. I only started after I noticed the contention on a production application where we are trying to use lucene in memory.
|Lots of commits here, a title that says 'lucene doesn't scale, lots of arguing that there are locking problems, but not one luceneutil benchmark run
??, You don't need to be a scientist nor use a profiling tool to see that all threads will block on a synchronized block or method on each request if that method is called during each search request, from that it doesn't take much to know that this does not scale.
Again, this is not a go at lucene, the heading 'Lucene Search not scalling' is causing toooo much problems here I see, I could have chosen a better heading and if we can change it to something else please do so. I think lucene is really great, thats why I'm trying to use it. Thanks for the support, ideas and patches so far.