Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Lucene's term vectors can now store payloads, but it looks like this was never exposed in Solr.

      1. SOLR-5911.patch
        24 kB
        David Smiley
      2. SOLR-5911.patch
        23 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch: I tried to find all the places that involved offsets with term vectors, and added a corresponding option for payloads. For FieldProperties I just assigned the next available flag; I assume I couldn't shift the existing ones (are these not supposed to change, for back-compat?).

        I also added returning payloads from TermVectorComponent, encoded as base64, and added a simple test confirming that payloads come back when indexed.

        Show
        Michael McCandless added a comment - Patch: I tried to find all the places that involved offsets with term vectors, and added a corresponding option for payloads. For FieldProperties I just assigned the next available flag; I assume I couldn't shift the existing ones (are these not supposed to change, for back-compat?). I also added returning payloads from TermVectorComponent, encoded as base64, and added a simple test confirming that payloads come back when indexed.
        Hide
        Michael McCandless added a comment -

        I think this patch is ready; I plan to commit it soon ... but if a dev more familiar w/ these parts of Solr could have a look, that would be great.

        Show
        Michael McCandless added a comment - I think this patch is ready; I plan to commit it soon ... but if a dev more familiar w/ these parts of Solr could have a look, that would be great.
        Hide
        Uwe Schindler added a comment -

        Move issue to Solr 4.9.

        Show
        Uwe Schindler added a comment - Move issue to Solr 4.9.
        Hide
        David Smiley added a comment -

        I reviewed the patch and brought it up to date with trunk. Nice thorough job Mike! At first I was just thinking this was a small matter of the schema but you thought of the TermVectorComponent, LukeRequestHandler, etc.

        I did make a change to TermVectorComponent.mapOneVector so that the postings flag indicates the options we want... and I simplified the code a little to not need the 3 useOffsets|Positions|Payloads booleans which seemed redundant with the same booleans on fieldOptions.

        Tests pass, precommit passes.

        If you don't have time to commit then I will be happy to.

        Show
        David Smiley added a comment - I reviewed the patch and brought it up to date with trunk. Nice thorough job Mike! At first I was just thinking this was a small matter of the schema but you thought of the TermVectorComponent, LukeRequestHandler, etc. I did make a change to TermVectorComponent.mapOneVector so that the postings flag indicates the options we want... and I simplified the code a little to not need the 3 useOffsets|Positions|Payloads booleans which seemed redundant with the same booleans on fieldOptions. Tests pass, precommit passes. If you don't have time to commit then I will be happy to.
        Hide
        ASF subversion and git services added a comment -

        Commit 1669492 from David Smiley in branch 'dev/trunk'
        [ https://svn.apache.org/r1669492 ]

        SOLR-5911: term vector payload support
        in schema & TermVectorComponent & LukeRequestHandler

        Show
        ASF subversion and git services added a comment - Commit 1669492 from David Smiley in branch 'dev/trunk' [ https://svn.apache.org/r1669492 ] SOLR-5911 : term vector payload support in schema & TermVectorComponent & LukeRequestHandler
        Hide
        ASF subversion and git services added a comment -

        Commit 1669494 from David Smiley in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1669494 ]

        SOLR-5911: term vector payload support
        in schema & TermVectorComponent & LukeRequestHandler

        Show
        ASF subversion and git services added a comment - Commit 1669494 from David Smiley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1669494 ] SOLR-5911 : term vector payload support in schema & TermVectorComponent & LukeRequestHandler
        Hide
        David Smiley added a comment -

        One additional slight change in the committed version is a comment to document termPayloads in the techproducts schema.xml (a kitchen sink schema) because it acts as documentation for the various options.

        Show
        David Smiley added a comment - One additional slight change in the committed version is a comment to document termPayloads in the techproducts schema.xml (a kitchen sink schema) because it acts as documentation for the various options.
        Hide
        Timothy Potter added a comment -

        Bulk close after 5.1 release

        Show
        Timothy Potter added a comment - Bulk close after 5.1 release

          People

          • Assignee:
            David Smiley
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development