Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When you index a field with IndexOptions.DOCS_AND_FREQS, Lucene will store just the docID and term frequency (how many times that term occurred in that document) for all documents that have a given term.

      We compute that term frequency by counting how many times a given token appeared in the field during analysis.

      But it can be useful, in expert use cases, to customize what Lucene stores as the term frequency, e.g. to hold custom scoring signals that are a function of term and document (this is my use case). Users have also asked for this before, e.g. see https://stackoverflow.com/questions/26605090/lucene-overwrite-term-frequency-at-index-time.

      One way to do this today is to stuff your custom data into a byte[] payload. But that's quite inefficient, forcing you to index positions, and pay the overhead of retrieving payloads at search time.

      Another approach is "token stuffing": just enumerate the same token N times where N is the custom number you want to store, but that's also inefficient when N gets high.

      I think we can make this simple to do in Lucene. I have a working version, using my own custom indexing chain, but the required changes are quite simple so I think we can add it to Lucene's default indexing chain?

      I created a new token attribute, TermDocFrequencyAttribute, and tweaked the indexing chain to use that attribute's value as the term frequency if it's present, and if the index options are DOCS_AND_FREQS for that field.

        Attachments

        1. LUCENE-7854.patch
          46 kB
          Michael McCandless
        2. LUCENE-7854.patch
          46 kB
          Michael McCandless
        3. LUCENE-7854.patch
          44 kB
          Michael McCandless
        4. LUCENE-7854.patch
          44 kB
          Michael McCandless
        5. LUCENE-7854.patch
          39 kB
          Michael McCandless
        6. LUCENE-7854.patch
          37 kB
          Michael McCandless
        7. LUCENE-7854.patch
          26 kB
          Michael McCandless

          Issue Links

            Activity

              People

              • Assignee:
                mikemccand Michael McCandless
                Reporter:
                mikemccand Michael McCandless
              • Votes:
                3 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: