So, good news / bad news...
The good news is I got a more mainstream test env (CentOS 5.4) online.
The bad news is the strange anomolies when testing NRT still occur,
and, flushing every 100 docs does not work around them.
But then the good news is, I managed to isolate the problem to the
hotspot compiler: somehow, it consistently compiles Lucene's search
code less efficiently (20-30% slower) depending on which test is being
run, which basically makes it impossible to really test performance
tradeoffs of NRT.
I've attached a simple SearchTest.java that shows the hotspot issue.
Run it like this:
java SearchTest /path/to/index <warmMethod>
I'm testing against a 5M doc Wikipedia index.
The <warmMethod> can be:
- "writer": open a writer, indexes docs, then rollback
- "nrt": same as "writer", but periodically get an NRT reader
- "reader": just open an IndexReader N times, then close it
- "searcher": same as "reader", but do some searching against each
After the warming, the test just runs a set of searches (TermQuery for
terms 0, 1, 2 ... 9) 10 times, then prints the min time.
I ran the tests on a 5M docs wikipedia index.
On nearly all JREs version I've tested, on OpenSolaris 2009.06 &
CentOS 5.4, warming with NRT causes a "permanent" loss of search
performance of somewhere between 20-30%. EG here's my results on
(In this case the "reader" warming also kicked hotspot into the slow
mode... it seems to be intermittant because sometimes "reader" is
I run java as "java -server -Xms1g -Xmx1g"
It's very odd... I suspect something buggy in hotspot, but I'm not
sure how to isolate it. It seems to somehow kick itself into a state
where it produces less optimal code for searching. And we're not
talking about that many methods, on the hotspots for running
I even printed out the assembly code (-XX:+PrintOptoAssembly) and it
was very strange – eg even IndexInput.readVInt was compiled
differently, if you warmed with "nrt" vs the others. I don't get it.
I'm trying to find a workaround that makes hotspot more manageable so
I can run real tests....