Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7583

Can we improve OutputStreamIndexOutput's byte buffering when writing each BKD leaf block?



    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 6.4, 7.0
    • None
    • None
    • New


      When BKD writes its leaf blocks, it's essentially a lot of tiny writes (vint, int, short, etc.), and I've seen deep thread stacks through our IndexOutput impl (OutputStreamIndexOutput) when pulling hot threads while BKD is writing.

      So I tried a small change, to have BKDWriter do its own buffering, by first writing each leaf block into a RAMOutputStream, and then dumping that (in 1 KB byte[] chunks) to the actual IndexOutput.

      This gives a non-trivial reduction (~6%) in the total time for BKD writing + merging time on the 20M NYC taxis nightly benchmark (2 times each):

      Trunk, sparse:

      • total: 64.691 sec
      • total: 64.702 sec

      Patch, sparse:

      • total: 60.820 sec
      • total: 60.965 sec

      Trunk dense:

      • total: 62.730 sec
      • total: 62.383 sec

      Patch dense:

      • total: 58.805 sec
      • total: 58.742 sec

      The results seem to be consistent and reproducible. I'm using Java 1.8.0_101 on a fast SSD on Ubuntu 16.04.

      It's sort of weird and annoying that this helps so much, because OutputStreamIndexOutput already uses java's BufferedOutputStream (default 8 KB buffer) to buffer writes.

      thetaphi suggested maybe hotspot is failing to inline/optimize the writeByte / the call stack just has too many layers.

      We could commit this patch (it's trivial) but it'd be nice to understand and fix why buffering writes is somehow costly so any other Lucene codec components that write lots of little things can be improved too.


        1. LUCENE-7583-hardcode-writeVInt.patch
          2 kB
          Michael McCandless
        2. LUCENE-7583.private-IndexOutput.patch
          2 kB
          Michael McCandless
        3. LUCENE-7583.patch
          4 kB
          Michael McCandless
        4. LUCENE-7583.patch
          32 kB
          Michael McCandless
        5. LUCENE-7583.fork-FastOutputStream.patch
          6 kB
          Michael McCandless



            Unassigned Unassigned
            mikemccand Michael McCandless
            0 Vote for this issue
            3 Start watching this issue