Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9458

WordDelimiterGraphFilter (and non-graph) should tie-break order using end offset

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.7
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      WordDelimiterGraphFilter and WordDelimiterFilter do not consult the end offset in their sub-token ordering.  In the event of a tie-break, I propose the longer token come first.  This usually happens already, but not always, and so this also feels like an inconsistency when you see it.  This issue can be thought of as a bug fix to LUCENE-9006 or an improvement; I have no strong feelings on the issue classification.  Before reading further, definitely read that issue.

      I see this is a problem when using CATENATE_ALL with either GENERATE_WORD_PARTS xor GENERATE_NUMBER_PARTS when the input ends with that part not being generated.  Consider the input: "other-9" and let's assume we want to catenate all, generate word parts, but nothing else (not numbers).  This should be tokenized in this order: "other9", "other" but today is emitted in reverse order.

       

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              dsmiley David Smiley
              Reporter:
              dsmiley David Smiley

              Dates

              • Created:
                Updated:
                Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 1h 10m
              1h 10m

                Issue deployment