Java NIO patch against Lucene 1.9

• Key: LUCENE-414
• Type: Bug
• Status: Resolved
• Resolution: Duplicate
• Priority: Minor
• Assignee: Unassigned
• Reporter: Chris Lamprecht
• Component/s: Store
• Affects Version/s: unspecified

Environment

Operating System: All
Platform: All

Description

Robert Engels previously submitted a patch against Lucene 1.4 for a Java NIO-
based Directory implementation. It also included some changes to FSDirectory
to allow better concurrency when searching from multiple threads. The
complete thread is at:

http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200505.mbox/%
[e-mail]%3e

This thread ended with Doug Cutting suggesting that someone port Robert's
changes to the SVN trunk. This is what I've done in this patch.

There are two parts to the patch. The first part modifies FieldsReader,
CompoundFileReader, and SegmentReader, to allow better concurrency when
reading an index. The second part includes the new NioFSDirectory
implementation, and makes small changes to FSDirectory and IndexInput to
accomodate this change. I'll put a more detailed outline of the changes to
each file in a separate message.

To use the new NioFSDirectory, set the system property
org.apache.lucene.FSDirectory.class to
org.apache.lucene.store.NioFSDirectory. This will cause
FSDirectory.getDirectory() to return an NioFSDirectory instance. By default,
NioFile limits the number of concurrent channels to 4, but you can override
this by setting the system property org.apache.lucene.nio.channels.

I did some performance tests with these patches. The biggest improvement came
from the concurrency improvements. NioFSDirectory performed about the same as
FSDirectory (with the concurrency improvements).

I ran my tests under Fedora Core 1; uname -a reports:
Linux myhost 2.4.22-1.2199.nptlsmp #1 SMP Wed Aug 4 11:48:29 EDT 2004 i686
i686 i386 GNU/Linux

The machine is a dual xeon 2.8GHz with 4GB RAM, and the tests were run against
a 9GB compound index file. The tests were run "hot" – with everything
already cached by linux's filesystem cache. The numbers are:

FSDirectory without patch: 13.3 searches per second
FSDirectory WITH concurrency patch: 14.3 searches per second

Both tests were run with 6 concurrent threads, which gave the highest numbers
in each case. I suspect that the concurrency improvements would make a bigger
difference on a more realistic test where the index isn't all cached in RAM
already, since the I/O happens whild holding the sychronized lock. Patches to
follow...

Thoughts?