Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5584

Allow FST read method to also recycle the output value when traversing FST

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.7.1
    • Fix Version/s: None
    • Component/s: core/FSTs
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The FST class heavily reuses Arc instances when traversing the FST. The output of an Arc however is not reused. This can especially be important when traversing large portions of a FST and using the ByteSequenceOutputs and CharSequenceOutputs. Those classes create a new byte[] or char[] for every node read (which has an output).
      In our use case we intersect a lucene Automaton with a FST<BytesRef> much like it is done in org.apache.lucene.search.suggest.analyzing.FSTUtil.intersectPrefixPaths() and since the Automaton and the FST are both rather large tens or even hundreds of thousands of temporary byte array objects are created.

      One possible solution to the problem would be to change the org.apache.lucene.util.fst.Outputs class to have two additional methods (if you don't want to change the existing methods for compatibility):

        /** Decode an output value previously written with {@link
         *  #write(Object, DataOutput)} reusing the object passed in if possible */
        public abstract T read(DataInput in, T reuse) throws IOException;
      
        /** Decode an output value previously written with {@link
         *  #writeFinalOutput(Object, DataOutput)}.  By default this
         *  just calls {@link #read(DataInput)}. This tries to  reuse the object   
         *  passed in if possible */
        public T readFinalOutput(DataInput in, T reuse) throws IOException {
          return read(in, reuse);
        }
      

      The new methods could then be used in the FST in the readNextRealArc() method passing in the output of the reused Arc. For most inputs they could even just invoke the original read(in) method.

      If you should decide to make that change I'd be happy to supply a patch and/or tests for the feature.

        Attachments

        1. fst-itersect-benchmark.tgz
          23 kB
          Christian Ziech

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              christianz Christian Ziech
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: