|
[
Permlink
| « Hide
]
Yonik Seeley added a comment - 19/Dec/06 06:37 PM
Patch for FSIndexInput to use a positional read call that doesn't use explicit synchronization. Note that the implementation of that read call may still involve some synchronization depending on the JVM and OS (notably Windows which lacks a native pread AFAIK).
This change should be faster on heavily loaded multi-threaded servers using the non-compound index format.
Performance tests are needed to see if there is any negative impact on single-threaded performance. Compound index format (CSIndexInput) still does synchronization because the base IndexInput is not cloned (and hence shared by all CSIndexInput clones). It's unclear if getting rid of the synchronization is worth the cloning overhead in this case. This patch continues to use BufferedIndexInput and allocates a new ByteBuffer for each call to read(). I wonder if it might be more efficient to instead directly extend IndexInput and always represent the buffer as a ByteBuffer?
CSIndexInput synchronization could also be elimitated if there was a pread added to IndexInput
public abstract void readBytes(byte[] b, int offset, int len, long fileposition) Unfortunately, that would break any custom Directory based implementations out there, and we can't provide a suitable default with seek & read because we don't know what object to synchronize on. Here is a patch that directly extends IndexInput to make things a little easier.
I started with the code for BufferedIndexInput to avoid any bugs in read(). They share enough code that a common subclass could be factored out if desired (or changes made in BufferedIndexInput to enable easier sharing). ByteBuffer does have offset, length, etc, but I did not use them because BufferedIndexInput currently allocates the byte[] on demand, and thus would add additional checks to readByte(). Also, the NIO Buffer.get() isn't as efficient as our own array access. You can find a NIO variation of IndexInput attached to this issue: http://issues.apache.org/jira/browse/LUCENE-519
I had good results on multiprocessor machines under heavy load. Regards, Thanks for the pointer Bogdan, it's interesting you use transferTo instead of read... is there any advantage to this? You still need to create a new object every read(), but at least it looks like a smaller object.
It's also been pointed out to me that http://issues.apache.org/jira/browse/LUCENE-414 The Javadoc says that transferTo can be more efficient because the OS can transfer bytes directly from the filesystem cache to the target channel without actually copying them.
> The Javadoc says that transferTo can be more efficient because the OS can transfer bytes
> directly from the filesystem cache to the target channel without actually copying them. Unfortunately, only for DirectByteBuffers and other FileChannels, not for HeapByteBuffers. Attaching test that reads a file in different ways, either random access or serially, from a number of threads.
Single-threaded random access performance of a fully cached 64MB file on my home PC (WinXP) , Java6:
config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=6518936 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=6518936 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=6518936 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=1024 filelen=6518936 Most of my workloads would benefit by removing the synchronization in FSIndexInput, so I took a closer look at this issue. I found exactly the opposite results that Yonik did on two platforms that I use frequently in production (Solaris and Linux), and by a significant margin. I even get the same behavior on the Mac, though I'm not running Java6 there.
config: impl=ChannelPread serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760
config: impl=ChannelPread serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760 config: impl=ClassicFile serial=false nThreads=200 iterations=10 bufsize=1024 filelen=10485760 Brad, one possible difference is the number of threads we tested with.
I tested single-threaded (nThreads=1) to see what kind of slowdown a single query might see. A normal production system shouldn't see 200 concurrent running search threads unless it's just about to fall over, or unless it's one of those massive multi-core systems. After you pass a certain amount of parallelism, NIO can help. Whoops; I should have paid more attention to the args. The results in the single-threaded case still favor pread, but by a slimmer margin:
Linux: config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 Solaris 10: config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 Mac OS X: config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=10485760 > Brad, [...]
That's Brian. And right, the difference in your tests is the number of threads. Perhaps this is a case where one size will not fit all. MmapDirectory is fastest on 64-bit platforms with lots of threads, while good-old-FSDirectory has always been fastest for single-threaded access. Perhaps a PreadDirectory would be the Directory of choice for multi-threaded access of large indexes on 32-bit hardware? It would be useful to benchmark this patch against MmapDirectory, since they both remove synchronization. My prior remarks were posted before I saw Brian's latest benchmarks.
While it would still be good to throw mmap into the mix, pread now looks like a strong contender for the one that might beat all. It works well on 32-bit hardware, it's unsynchronized, and it's fast. What's not to like? Weird... I'm still getting slower results from pread on WinXP.
Can someone else verify on a windows box? Yonik@spidey ~ $ c:/opt/jdk16/bin/java -server FileReadTest testfile ClassicFile false 1 200 config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=9616000 answer=160759732, ms=14984, MB/sec=128.35024025627337 $ c:/opt/jdk16/bin/java -server FileReadTest testfile ClassicFile false 1 200 config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=9616000 answer=160759732, ms=14640, MB/sec=131.36612021857923 $ c:/opt/jdk16/bin/java -server FileReadTest testfile ChannelPread false 1 200 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=9616000 answer=160759732, ms=21766, MB/sec=88.35798952494717 $ c:/opt/jdk16/bin/java -server FileReadTest testfile ChannelPread false 1 200 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=9616000 answer=160759732, ms=21718, MB/sec=88.55327378211622 $ c:/opt/jdk16/bin/java -version java version "1.6.0_02" Java(TM) SE Runtime Environment (build 1.6.0_02-b06) Java HotSpot(TM) Client VM (build 1.6.0_02-b06, mixed mode) I sent this via email, but probably need to add to the thread...
I posted a bug on this to Sun a long while back. This is a KNOWN BUG on Windows. NIO preads actually sync behind the scenes on some platforms. Using multiple file descriptors is much faster. See bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734 So it looks like pread is ~50% slower on Windows, and ~5-25% faster on other platforms. Is that enough of a difference that it might be worth having FSDirectory use different implementations of FSIndexInput based on the value of Constants.WINDOWS (and perhaps JAVA_VERSION)?
+1 I think having good out-of-the-box defaults is extremely important (most users won't tune), and given the substantial cross platform differences here I think we should conditionalize the defaults according to the platform. As an aside, if the Lucene people voted on the Java bug (and or sent emails via the proper channels), hopefully the underlying bug can be fixed in the JVM.
In my opinion it is a serious one - ruins any performance gains of using NIO on files. Updated test that fixes some thread synchronization issues to ensure that the "answer" is the same for all methods.
Brian, in some of your tests the answer is "0"... is this because your test file consists of zeros (created via /dev/zero or equiv)? UNIX systems treat blocks of zeros differently than normal files (they are stored as holes). It shouldn't make too much of a difference in this case, but just to be sure, could you try with a real file? Yeah, the file was full of zeroes. But I created the files w/o holes and was using filesystems that don't compress file contents. Just to be sure, though, I repeated the tests with a file with random contents; the results above still hold.
BTW, I think the performance win with Yonik's patch for some workloads could be far greater than what the simple benchmark illustrates. Sure, pread might be marginally faster. But the real win is avoiding synchronized access to the file.
I did some IO tracing a while back on one particular workload that is characterized by:
In this workload where each query hits each compound index, the locking in FSIndexInput means that a single rare query clobbers the response time for all queries. The requests to read cached data are serialized (fairly, even) with those that hit the disk. While we can't get rid of the rare queries, we can allow the common ones to proceed against cached data right away. I ran Yonik's most recent FileReadTest.java on the platforms below,
testing single-threaded random access for fully cached 64 MB file. I tested two Windows XP Pro machines and got opposite results from I'm showing ChannelTransfer to be much faster on all platforms except The ChannelTransfer test is giving the wrong checksum, but I think Mac OS X 10.4 (Sun java 1.5) config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 Linux 2.6.22.1 (Sun java 1.5) config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 Windows Server 2003 R2 Enterprise x64 (Sun java 1.6) config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 Windows XP Pro SP2, laptop (Sun Java 1.5) config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 Windows XP Pro SP2, older desktop (Sun Java 1.6) config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=6518936 filelen=67108864 I also just ran a test with 4 threads, random access, on Linux 2.6.22.1:
config: impl=ClassicFile serial=false nThreads=4 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelFile serial=false nThreads=4 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelPread serial=false nThreads=4 iterations=200 bufsize=6518936 filelen=67108864 config: impl=ChannelTransfer serial=false nThreads=4 iterations=200 bufsize=6518936 filelen=67108864 ChannelTransfer got even faster (scales up with added threads better). Mike, it looks like you are running with a bufsize of 6.5MB!
Apologies for my hard-to-use benchmark program  I'll try fixing the transferTo test before anyone re-runs any tests.
Doh!! Woops
 I will rerun... OK, uploading latest version of the test that should fix ChannelTransfer (it's also slightly optimized to not create a new object per call).
Well, at least we've learned that printing out all the input params for benchmarking programs is good practice Thanks! I'll re-run.
Yes indeed OK my results on Win XP now agree with Yonik's.
On UNIX & OS X, ChannelPread is a bit (2-14%) better, but on windows Win Server 2003 R2 Enterprise x64 (Sun Java 1.6): config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=68094, MB/sec=197.10654095808735 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=72594, MB/sec=184.88818359644048 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=98328, MB/sec=136.5000081360345 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=201563, MB/sec=66.58847506734867 Win XP Pro SP2, laptop (Sun Java 1.5): config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=47449, MB/sec=282.8673481000653 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=54899, MB/sec=244.4811890926975 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=71683, MB/sec=187.237877878995 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=149475, MB/sec=89.79275999330991 Linux 2.6.22.1 (Sun Java 1.5): config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=41162, MB/sec=326.0719304212623 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=53114, MB/sec=252.69745829724744 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=40226, MB/sec=333.65914582608264 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=59163, MB/sec=226.86092321214272 Mac OS X 10.4 (Sun Java 1.5): config: impl=ClassicFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=85894, MB/sec=156.25972477705076 config: impl=ChannelFile serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=109939, MB/sec=122.08381738964336 config: impl=ChannelPread serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=75517, MB/sec=177.73180608339845 config: impl=ChannelTransfer serial=false nThreads=1 iterations=200 bufsize=1024 filelen=67108864 answer=110480725, ms=130156, MB/sec=103.12066136021389 lucene-753.patch
Made NIOFSDirectory that inherits from FSDirectory and includes the patch. Carrying forward from this thread: Jason Rutherglen <jason.rutherglen@gmail.com> wrote:
It wasn't clear to me that pread would in fact perform better than So I modified (attached) FileReadTest.java to add a new SeparateFile The results are very interesting – using SeparateFile is always I don't have a Windows server class machine readily accessible so if This is a strong argument for some sort of pooling of Mac OS X 10.5.3, single WD Velociraptor hard drive, Sun JRE 1.6.0_05 config: impl=ClassicFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=151884, MB/sec=176.73715203708093 config: impl=SeparateFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=97820, MB/sec=274.4177632386015 config: impl=ChannelPread serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=103059, MB/sec=260.4677476008888 config: impl=ChannelFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=176250, MB/sec=152.30380482269504 config: impl=ChannelTransfer serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=365904, MB/sec=73.36226332589969 Linux 2.6.22.1, 6-drive RAID 5 array, Sun JRE 1.6.0_06 config: impl=ClassicFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=75592, MB/sec=355.1109323737962 config: impl=SeparateFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=35505, MB/sec=756.0497282072947 config: impl=ChannelPread serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=51075, MB/sec=525.5711326480665 config: impl=ChannelFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=95640, MB/sec=280.6727896277708 config: impl=ChannelTransfer serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=93711, MB/sec=286.45031639828835 WIN XP PRO, laptop, Sun JRE 1.4.2_15: config: impl=ClassicFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=135349, MB/sec=198.32836297275932 config: impl=SeparateFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=62970, MB/sec=426.2910211211688 config: impl=ChannelPread serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=174606, MB/sec=153.73781886074937 config: impl=ChannelFile serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=152171, MB/sec=176.4038193873997 config: impl=ChannelTransfer serial=true nThreads=4 iterations=100 bufsize=1024 filelen=67108864 answer=-23909200, ms=275603, MB/sec=97.39932293915524 Interesting results. The question would be, what would the algorithm for allocating RandomAccessFiles to which file look like? When would a new file open, when would a file be closed? If it is based on usage would it be based on the rate of calls to readInternal? This seems like an OS filesystem topic that maybe there is some standard algorithm for. How would the pool avoid the same synchronization issue given the default small buffer size of 1024? If there are 30 threads executing searches, there will not be 30 RandomAccessFiles per file so there is still contention over the limited number of RandomAccessFiles allocated.
Added a PooledPread impl to FileReadTest, but at least on Windows it always seems slower than non-pooled. I suppose it might be because of the extra synchronization.
I think you have a small bug – minCount is initialized to 0 but should be something effectively infinite instead?
Thanks Mike, after the bug is fixed, PooledPread is now faster on Windows when more than 1 thread is used.
OK I re-ran only PooledPread, SeparateFile and ChannelPread since they
are the "leading contenders" on all platforms. Also, I changed to serial=false. Now the results are very close on all but windows, but on windows I'm Mac OS X 10.5.3, single WD Velociraptor hard drive, Sun JRE 1.6.0_05 config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=120807, MB/sec=222.20190551871994 config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830326, ms=119641, MB/sec=224.36744594244448 config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=119217, MB/sec=225.1654176837196 Linux 2.6.22.1, 6-drive RAID 5 array, Sun JRE 1.6.0_06 config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=52613, MB/sec=510.2074696367818 config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=52715, MB/sec=509.22025230010433 config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=53792, MB/sec=499.0248661511005 WIN XP PRO, laptop, Sun JRE 1.4.2_15: config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=209956, MB/sec=127.85319590771401 config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=89101, MB/sec=301.27098012367986 config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864 answer=-23830370, ms=184087, MB/sec=145.81988733587923 I was curious about the discrepancy between the ChannelPread implementation and the SeparateFile implementation that Yonik saw. At least on Mac OS X, the kernel implementation of read is virtually the same as pread, so there shouldn't be any appreciable performance difference unless the VM is doing something funny. Sure enough, the implementations of read() under RandomAccessFile and read() under FileChannel are totally different. The former relies on a buffer allocated either on the stack or by malloc, while the latter allocates a native buffer and copies the results to the original array.
Switching to a native buffer in the benchmark yields identical results for ChannelPread and SeparateFile on 1.5 and 1.6 on OS X. I'm attaching an implementation of ChannelPreadDirect that uses a native buffer. This may be a moot point because any implementation inside Lucene needs to consume a byte[] and not a ByteBuffer, but at least it's informative. Here are some of my results with 4 threads and a pool size of 4 fds per file. Win XP on a Pentium4 w/ Java5_0_11 -server
config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=9616000 answer=322211190, ms=51891, MB/sec=74.12460735002217 config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=9616000 answer=322211190, ms=71175, MB/sec=54.04144713733755 config: impl=ClassicFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=9616000 answer=322211190, ms=62699, MB/sec=61.34707092617107 config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=9616000 answer=322211410, ms=21324, MB/sec=180.37891577565185 OK it's looking like SeparateFile is the best overall choice... it It's somewhat surprising to me that after all this time, with these Of course this is a synthetic benchmark. Actual IO with Lucene is
Ideally it would be based roughly on contention. EG a massive CFS I think it would have thread affinity (if the same thread wants the
I think this should be reference counting. The first time Lucene
Fortunately, Lucene tends to call IndexInput.clone() when it wants to So I think the pool could work something like this: FSIndexInput.clone One problem with this approach is I'm not sure clones are always An alternative approach would be to sync() on every block (1024 bytes In fact, if we build this pool, we can again try all these alternative As I stated quit a while ago, this has been a long accepted bug in the JDK.
See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734 It was filed and accepted over 3 years ago. The problem is that the pread performs an unnecessary lock on the file descriptor, instead of using Windows "overlapped" reads. The point being - please vote for this issue so it can be fixed properly. It is really a trivial fix, but it needs to be done by SUN.
.bq OK it's looking like SeparateFile is the best overall choice... it matches the best performance on Unix platforms and is very much the
lead on Windows. The other implementations are fully-featured though (they could be used in lucene w/ extra synchronization, etc). SeparateFile (opening a new file descriptor per reader) is not a real implementation that could be used... it's more of a theoretical maximum IMO. Also remember that you can't open a new fd on demand since the file might already be deleted. We would need a real PooledClassicFile implementation (like PooledPread). On non-windows it looks like ChannelPread is probably the right choice.. near max performance and min fd usage Core2Duo Windows XP JDK1.5.15. PooledPread for 4 threads and pool size 2 the performance does not compare well to SeparateFile. PooledPread for 30 threads does not improve appreciably over ClassicFile. If there were 30 threads, how many RandomAccessFiles would there need to be to make a noticeable impact? The problem I see with the pooled implementation is setting the global file descriptor limit properly, will the user set this? There would almost need to be a native check to see if what the user is trying to do is possible.
The results indicate there is significant contention in the pool code. The previous tests used a pool size the same as the number of threads which is probably not how most production systems look, at least the SOLR installations I've worked on. In SOLR the web request thread is the thread that executes the search, so the number of threads is determined by the J2EE server which can be quite high. Unless the assumption is the system is set for an unusually high number of file descriptors. There should probably be a MMapDirectory test as well. config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=53797223, ms=32715, MB/sec=221.4329573590096 config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=53797223, ms=18687, MB/sec=387.6587574249478 config: impl=SeparateFile serial=false nThreads=30 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=403087371, ms=137871, MB/sec=394.0737646060448 config: impl=PooledPread serial=false nThreads=30 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=403087487, ms=526504, MB/sec=103.19265190767781 config: impl=ChannelPread serial=false nThreads=30 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=403087487, ms=624291, MB/sec=87.02887595688549 config: impl=ClassicFile serial=false nThreads=30 iterations=100 bufsize=1024 poolsize=2 filelen=18110448 answer=403087487, ms=587430, MB/sec=92.48990347786119 config: impl=PooledPread serial=false nThreads=30 iterations=100 bufsize=1024 poolsize=4 filelen=18110448 answer=403087487, ms=552971, MB/sec=98.25351419875544
True, we'd have to make a real pool, but I'd think we want the sync() to be on cloning and not on every read. I think Lucene's usage of the open files (clones are made & used up quickly and closed) would work well with that approach. I think at this point we should build out an underlying pool and then test all of these different approaches under the pool. And yes we cannot just open a new fd on demand if the file has been deleted. But I'm thinking that may not matter in practice. Ie if the pool wants to open a new fd, it can attempt to do so, and if the file was deleted it must then return a shared access wrapper to the fd it already has open. Large segments are where the contention will be and large segments are not often deleted. Plus people tend to open new readers if such a large change has taken place to the index.
But on Linux I saw 44% speedup for serial=true case with 4 threads using SeparateFile vs ChannelPread, which I was very surprised by. But then again it's synthetic so it may not matter in real Lucene searches. lucene-753.patch
Added javadoc and removed unnecessary NIOFSIndexOutput class.
IIRC, clones are often not closed at all.
At first blush, sounds a bit too complex for the benefits.
In the serial case, there are half the system calls (no seek). When both implementations have a single single system call, all the extra code and complexity that Sun threw into FileChannel comes into play. Compare that with RandomAccessFile.read() which drops down to a native call and presumably just the read with little overhead. I wish Sun would just add a RandomAccessFile.read with a file position. If access will be truly serial sometimes, larger buffers would help with that larger read() setup cost.
Right but that'd all be under one thread right? The pool would always give the same RandomAccessFile (private or shared) for the same filename X thread.
Well, I think you'd hand it out first, as a shared file (so you reserve the right to hand it out again, later). If other threads come along you would open a new one (if you are under the budget) and loan it to them privately (so no syncing during read). I think sync'ing with no contention (the first shared file we hand out) should be OK performance in modern JVMs.
But not on Windows...
Yeah I'm on the fence too ... but this lack of concurrency hurts our search performance. It's ashame users have to resort to multiple IndexReaders. Though it still remains to be seen how much the pool or pread approaches really improve end to end search performance (vs other bottlenecks like IndexReader.isDeleted). Windows is an important platform and doing the pool approach, vs leaving Windows with classic if we do pread approach, lets us have good concurrency on Windows too. This probably doesn't help much, but I implemented a pool and submitted a patch very similar to the SeparateFile approach. Before being directed to this thread:
https://issues.apache.org/jira/browse/LUCENE-1337
In our implementation the synchronization/lack of concurrency has been a big issue for us. On several occasions we've had to remove new features that perform searches from frequently hit pages, because threads build up waiting for synchronized access to the underlying files. It is possible that I would still have issue even with my patch, considering from my tests that I'm only increasing throughput by 300%, but it would be easier for me to tune and scale my application since resource utilization and contention would be visible from the OS level. > At first blush, sounds a bit too complex for the benefits. My vote is that the benefits outway the complexity, especially considering it's an out-of-the box solutions that works well for all platforms and single threaded as well as multi-threaded envirnments. If it's helpful, I can spend the time to implement some of the missing feature(s) of the pool that will be needed for it to be an acceptable solution (i.e, shared access once a file has been deleted, and perhaps a time-based closing mechanism).
Can you describe your test – OS, JRE version, size/type of your index, number of cores, amount of RAM, type of IO system, etc? It's awesome that you see 300% gain in search throughput. Is your index largely cached in the OS's IO cache, or not?
If we can see sizable concurreny gains, reliably & across platforms, I agree we should pursue this approach. One particular frustration is: if you optimize your index, thinking this gains you better search performance, you're actually making things far worse as far as concurrency is concerned because now you are down to a single immense file. I think we do need to fix this situation. On your patch, I think in addition to shared-access on a now-deleted file, we should add a global control on the "budget" of number of open files (right now I think your patch has a fixed cap per-filename). Probably the budget should be expressed as a multiplier off the minimum number of open files, rather than a fixed cap, so that an index with many segments is allowed to use more. Ideally over time the pool works out such that for small files in the index (small segments) since there is very little contention they only hold 1 descriptor open, but for large files many descriptors are opened. I created a separate test (will post a patch & details to this issue) to explore using SeparateFile inside FSDirectory, but unfortunately I see mixed results on both the cached & uncached cases. I'll post details separately. One issue with your patch is it's using Java 5 only classes (Lucene is still on 1.4); once you downgrade to 1.4 I wonder if the added synchronization will become costly. I like how your approach is to pull a RandomAccessFile from the pool only when a read is taking place – this automatically takes care of creating new descriptors when there truly is contention. But one concern I have is that this defeats the OS's IO system's read-ahead optimization since from the OS's perspective the file descriptors are getting shuffled. I'm not sure if this really matters much in Lucene, because many things (reading stored fields & term vectors) are likely not helped much by read-ahead, but for example a simple TermQuery on a large term should in theory benefit from read-ahead. You could gain this back with a simple thread affinity, such that the same thread gets the same file descriptor it got last time, if it's available. But that added complexity may offset any gains. I attached FSDirectoryPool.patch, which adds This is intended only as a test (to see if shows consistent I also added a "pool=true|false" config option to contrib/benchmark so I ran some quick initial tests but didn't see obvious gains. I'll go I created a large index (indexed Wikipedia 4X times over, with stored
fields & tv offsets/positions = 72 GB). I then randomly sampled 50 terms > 1 million freq, plus 200 terms > 100,000 freq plus 100 terms > 10,000 freq plus 100 terms > 1000 freq. Then I warmed the OS so these queries are fully cached in the IO cache. It's a highly synthetic test. I'd really love to test on real Then I ran this alg: analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer query.maker = org.apache.lucene.benchmark.byTask.feeds.FileBasedQueryMaker file.query.maker.file = /lucene/wikiQueries.txt directory=FSDirectory pool=true work.dir=/lucene/bigwork OpenReader { "Warmup" SearchTrav(20) > : 5 { "Rounds" [{ "Search" Search > : 500]: 16 NewRound }: 2 CloseReader RepSumByPrefRound Search I ran with 2, 4, 8 and 16 threads, on a Intel quad Mac Pro (2 cpus, Here're the results – each run is best of 2, and all searches are
I also ran the same alg, replacing Search task with SearchTravRet(10)
So there are smallish gains, but rememember these are upper bounds on OK I ran the uncached test, using the Search task. JRE & hardware are I generated a larger (6150) set of queries to make sure the threads
The gains are better. The 8 thread case I don't get; I re-ran it and I just tried out the latest NIOFSDirectory patch and I'm seeing a bug. If I go back to the regular FSDirectory, everything works fine.
I can't reproduce it on a smaller testcase. It only happens with the live index. Any ideas on where to debug? Caused by: java.lang.IndexOutOfBoundsException: Index: 24444, Size: 4 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:260) at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:249) at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:68) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123) at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154) at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678) at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87) at org.apache.lucene.search.Similarity.idf(Similarity.java:457) at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362) at org.apache.lucene.search.Query.weight(Query.java:95) at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171) at org.apache.lucene.search.Searcher.search(Searcher.java:132) The index is not using the compound file format:
7731499698 Jul 28 03:46 _6zk.fdt
232014520 Jul 28 03:50 _6zk.fdx
32 Jul 28 03:50 _6zk.fnm
3775713450 Jul 28 04:06 _6zk.frq
58003634 Jul 28 04:07 _6zk.nrm
2944298834 Jul 28 04:18 _6zk.prx
432418 Jul 28 04:18 _6zk.tii
30784106 Jul 28 04:19 _6zk.tis
217354711 Jul 28 08:18 _76i.fdt
6509864 Jul 28 08:18 _76i.fdx
32 Jul 28 08:18 _76i.fnm
144348761 Jul 28 08:18 _76i.frq
1627470 Jul 28 08:18 _76i.nrm
295528445 Jul 28 08:19 _76i.prx
52622 Jul 28 08:19 _76i.tii
3858378 Jul 28 08:19 _76i.tis
199621206 Jul 29 13:29 _7cm.fdt
5994720 Jul 29 13:29 _7cm.fdx
32 Jul 29 13:29 _7cm.fnm
136445620 Jul 29 13:29 _7cm.frq
1498684 Jul 29 13:29 _7cm.nrm
284805312 Jul 29 13:30 _7cm.prx
48346 Jul 29 13:30 _7cm.tii
3522117 Jul 29 13:30 _7cm.tis
3914068 Jul 29 13:30 _7cn.fdt
119184 Jul 29 13:30 _7cn.fdx
32 Jul 29 13:30 _7cn.fnm
2993343 Jul 29 13:30 _7cn.frq
29800 Jul 29 13:30 _7cn.nrm
7380878 Jul 29 13:30 _7cn.prx
5277 Jul 29 13:30 _7cn.tii
378816 Jul 29 13:30 _7cn.tis
383147 Jul 29 13:30 _7cq.fdt
11240 Jul 29 13:30 _7cq.fdx
32 Jul 29 13:30 _7cq.fnm
290398 Jul 29 13:30 _7cq.frq
2814 Jul 29 13:30 _7cq.nrm
763135 Jul 29 13:30 _7cq.prx
1581 Jul 29 13:30 _7cq.tii
115971 Jul 29 13:30 _7cq.tis
19 Jul 29 13:30 date
20 Jul 21 01:53 segments.gen
155 Jul 29 13:30 segments_d61
Is the index itself corrupt, ie, NIOFSDirectory did something bad when writing the index? Or, is it only in reading the index with NIOFSDirectory that you see this? IE, can you swap in FSDirectory on your existing index and the problem goes away?
I haven't seen any issues with writing the index under NIOFSDirectory. The failures seem to happen only when reading. When I switch to FSDirectory (or MMapDirectory), the same index that fails under NIOFSDirectory works flawlessly (indicating that the index is not corrupt). The error with NIOFSDirectory is determinate and repeatable (same error every time, same location, same query during warmup). I couldn't reproduce this on a smaller index, unfortunately.
Did you see a prior exception, before hitting the AIOOBE? If so, I think this is just I can possibly work on this, just go through and reedit the BufferedIndexInput portions of the code. Inheriting is difficult because of the ByteBuffer code. Needs to be done line by line.
Attached new rev of NIOFSDirectory.
Besides re-fixing Matthew, could you give this one a shot to see if it fixes your case? Thanks.
That would be awesome, Jason. I think we should then commit NIOFSDirectory to core as at least a way around this bottleneck on all platforms but Windows. Maybe we can do this in time for 2.4?
Michael, I ran this new patch against our big index and it works very well. If I have time, I'll run some benchmarks to see what our real-life performance improvements are like. Note that I'm only running it for our read-only snapshot of the index, however, so this hasn't been tested for writing to a large index.
Super, thanks! This was the same index that would reliably hit the exception above?
Correct - it would hit the exception every time at startup. I've been running NIOFSDirectory for the last couple of hours with zero exceptions (except for running out of file descriptors after starting it the first time Thanks for all the work on this patch. This exception popped up out of the blue a few hours in. No exceptions before it. I'll see if I can figure out whether it was caused by our index snapshotting or if it's a bug elsewhere in NIOFSDirectory.
I haven't seen any exceptions like this with MMapDirectory, but it's possible there's something that we're doing that isn't correct. Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616) at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.read(NIOFSDirectory.java:186) at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.refill(NIOFSDirectory.java:218) at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.readByte(NIOFSDirectory.java:232) at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123) at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154) at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678) at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373) at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:310) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87) at org.apache.lucene.search.Searcher.docFreqs(Searcher.java:178) Interesting...
Are you really sure you're not accidentally closing the searcher before calling Searcher.docFreqs? Are you calling docFreqs directly from your app? It looks like MMapIndexInput.close() is a noop so it would not have detected calling Searcher.docFreqs after close, whereas NIOFSdirectory (and the normal FSDirectory) will. If you try the normal FSDirectory do you also an exception like this? Incidentally, what sort of performance differences are you noticing between these three different ways of accessing an index in the file system?
+1 Latest patch is looking good to me! Also, our finalizers aren't technically thread safe which could lead to a double close in the finalizer (although I doubt if this particular case would ever happen). If we need to keep them, we could change Descriptor.isOpen to volatile and there should be pretty much no cost since it's only checked in close().
Yonik, do you mean BufferedIndexInput.clone (not FSIndexInput)? I think once we fix NIOFSIndexInput to subclass from BufferedIndexInput, then cloning should be lazy again. Jason are you working on this (subclassing from BufferedIndexInput)? If not I can take it.
Hmmm... I'll update both FSDirectory and NIOFSDiretory's isOpen's to be volatile. Mike I have have not started on the subclassing from BufferedIndexInput yet. I can work on it monday though.
OK, thanks! Updated patch with Yonik's volatile suggestion – thanks Yonik!
Also, I removed NIOFSDirectory.createOutput since it was doing the same thing as super(). Michael,
Our IndexReaders are actually managed in a shared pool (currently 8 IndexReaders, shared round-robin style as requests come in). We have some custom reference counting logic that's supposed to keep the readers alive as long as somebody has them open. As new index snapshots come in, the IndexReaders are re-opened and reference counts ensure that any old index readers in use are kept alive until the searchers are done with them. I'm guessing we have an error in our reference counting logic that just doesn't show up under MMapDirectory (as you mentioned, close() is a no-op). We're calling docFreqs directly from our app. I'm guessing that it just happens to be the most likely item to be called after we roll to a new index snapshot. I don't have hard performance numbers right now, but we were having a hard time saturating I/O or CPU with FSDirectory. The locking was basically killing us. When we switched to MMapDirectory and turned on compound files, our performance jumped at least 2x. The preliminary results I'm seeing with NIOFSDirectory seem to indicate that it's slightly faster than MMapDirectory. I'll try setting our app back to using the old FSDirectory and see if the exceptions still occur. I'll also try to fiddle with our unit tests to make sure we're correctly ref-counting all of our index readers. BTW, I ran a quick FSDirectory/MMapDirectory/NIOFSDirectory shootout. It uses a parallel benchmark that roughly models what our real-life benchmark is like. I ran the benchmark once through to warm the disk cache, then got the following. The numbers are fairly stable across various runs once the disk caches are warm: FS: 33644ms I'm a bit surprised at the results myself, but I've spent a bit of time tuning the indexes to maximize concurrency. I'll double-check that the benchmark is correctly running all of the tests. The benchmark effectively runs 10-20 queries in parallel at a time, then waits for all queries to complete. It does this end-to-end for a number of different query batches, then totals up the time to complete each batch. LUCENE-753.patch
NIOFSIndexInput now extends BufferedIndexInput. I was unable to test however and wanted to just get this up.
This is surprising – your benchmark is very concurrent, yet FSDir and NIOFSDir are close to the same net throughput, while MMapDir is quite a bit faster. Is this on a non-Windows OS? New patch attached. Matthew if you could try this version out on your I didn't like how we were still copying the hairy readBytes & refill But, then I realized we were duplicating alot of code from Some other things also fixed:
To test this, I made NIOFSDirectory the default IMPL in I also built first 150K docs of wikipedia and ran various searches The class is quite a bit simpler now, however there's one thing I New version attached. This one re-uses a wrapped byte buffer even when it's CSIndexInput that's calling it.
I plan to commit in a day or two. I just committed revision 690539, adding NIOFSDirectory. I will leave this open, but move off of 2.4, until we can get similar performance gains on Windows...
SUN is accepting outside bug fixes to the Open JDK, and merging them to the commercial JDK (in most cases).
If the underlying bug is fixed in the Windows JDK - not too hard - then you fix this properly in Lucene. If you don't fix it in the JDK you are always going to have the 'running out of file handles' synchronization, vs, the "locked position" synchronization - there is no way to fix this in user code... Attaching new FileReadTest.java that fixes a concurrency bug in SeparateFile - each reader needed it's own file position.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
New Feature
Open
Major
Use NIO positional read to avoid synchronization in FSIndexInput
