Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9458

WordDelimiterGraphFilter (and non-graph) should tie-break order using end offset

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.7
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      WordDelimiterGraphFilter and WordDelimiterFilter do not consult the end offset in their sub-token ordering.  In the event of a tie-break, I propose the longer token come first.  This usually happens already, but not always, and so this also feels like an inconsistency when you see it.  This issue can be thought of as a bug fix to LUCENE-9006 or an improvement; I have no strong feelings on the issue classification.  Before reading further, definitely read that issue.

      I see this is a problem when using CATENATE_ALL with either GENERATE_WORD_PARTS xor GENERATE_NUMBER_PARTS when the input ends with that part not being generated.  Consider the input: "other-9" and let's assume we want to catenate all, generate word parts, but nothing else (not numbers).  This should be tokenized in this order: "other9", "other" but today is emitted in reverse order.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dsmiley David Smiley
                Reporter:
                dsmiley David Smiley
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m