Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-4620

Explore IntEncoder/Decoder bulk API


    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.1, 6.0
    • modules/facet
    • None
    • New, Patch Available


      Today, IntEncoder/Decoder offer a streaming API, where you can encode(int) and decode(int). Originally, we believed that this layer can be useful for other scenarios, but in practice it's used only for writing/reading the category ordinals from payload/DV.

      Therefore, Mike and I would like to explore a bulk API, something like encode(IntsRef, BytesRef) and decode(BytesRef, IntsRef). Perhaps the Encoder can still be streaming (as we don't know in advance how many ints will be written), dunno. Will figure this out as we go.

      One thing to check is whether the bulk API can work w/ e.g. facet associations, which can write arbitrary byte[], and so may decoding to an IntsRef won't make sense. This too we'll figure out as we go. I don't rule out that associations will use a different bulk API.

      At the end of the day, the requirement is for someone to be able to configure how ordinals are written (i.e. different encoding schemes: VInt, PackedInts etc.) and later read, with as little overhead as possible.


        1. LUCENE-4620.patch
          37 kB
          Shai Erera
        2. LUCENE-4620.patch
          3 kB
          Michael McCandless
        3. LUCENE-4620.patch
          1 kB
          Michael McCandless
        4. LUCENE-4620.patch
          244 kB
          Shai Erera
        5. LUCENE-4620.patch
          239 kB
          Shai Erera
        6. LUCENE-4620.patch
          250 kB
          Shai Erera



            shaie Shai Erera
            shaie Shai Erera
            0 Vote for this issue
            1 Start watching this issue