Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10349

Add totalTermFreq support to TermsComponent

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.6, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      See discussion here: http://markmail.org/message/gmpmege2jpfrsp75. Both docFreq and totalTermFreq are already available to the TermsComponent, it's just that doesn't add the ttf measure to the response.

      This issue adds a new terms.ttf parameter which if set to true results in the following output:

      <lst name="terms">
        <lst name="standardfilt">
          <lst name="shark">
            <long name="docFreq">2</long>
            <long name="totalTermFreq">2</long>
          </lst>
      ...
      

      The reason for the new parameter is to not break backward-compatibility, though I wish we could always return those two measures (it doesn't cost us anything, the two are already available to the code). Maybe we can break the response in master and add this parameter only to 6x as deprecated? I am also fine if we leave it and handle it in a separate issue.

      1. SOLR-10349.patch
        17 kB
        Shai Erera
      2. SOLR-10349.patch
        9 kB
        Shai Erera
      3. SOLR-10349.patch
        8 kB
        Shai Erera

        Activity

        Hide
        shaie Shai Erera added a comment -

        Patch implements the proposed addition. Joel Bernstein, not sure if you're still interested reviewing this, but if you are, your comments are appreciated!

        Show
        shaie Shai Erera added a comment - Patch implements the proposed addition. Joel Bernstein , not sure if you're still interested reviewing this, but if you are, your comments are appreciated!
        Hide
        shaie Shai Erera added a comment -

        Added CHANGES entry.

        Show
        shaie Shai Erera added a comment - Added CHANGES entry.
        Hide
        mkhludnev Mikhail Khludnev added a comment -

        +1

        Show
        mkhludnev Mikhail Khludnev added a comment - +1
        Hide
        joel.bernstein Joel Bernstein added a comment -

        Looks good, maybe just needs a distributed test?

        I agree this makes sense in the default response. Do we think adding this new field will cause problems with the client? Does this just add a field or has the response structure changed?

        Show
        joel.bernstein Joel Bernstein added a comment - Looks good, maybe just needs a distributed test? I agree this makes sense in the default response. Do we think adding this new field will cause problems with the client? Does this just add a field or has the response structure changed?
        Hide
        shaie Shai Erera added a comment -

        Thanks Joel Bernstein, the distributed test suggestion helped me find DistributedTermsComponentTest, and of course as soon as I added a test to it, the client failed. Since it expects a number, but got a map. I will see how to fix it.

        This also answers your second question, this commit changes the response structure if you ask for terms.ttf. I put an example output in the description above.

        Show
        shaie Shai Erera added a comment - Thanks Joel Bernstein , the distributed test suggestion helped me find DistributedTermsComponentTest , and of course as soon as I added a test to it, the client failed. Since it expects a number, but got a map. I will see how to fix it. This also answers your second question, this commit changes the response structure if you ask for terms.ttf . I put an example output in the description above.
        Hide
        shaie Shai Erera added a comment -

        That was a good comment Joel Bernstein!! I changed more code to adapt the new format when necessary. Running tests now, but if you think/know of other places which might be affected by this change, please let me know.

        Show
        shaie Shai Erera added a comment - That was a good comment Joel Bernstein !! I changed more code to adapt the new format when necessary. Running tests now, but if you think/know of other places which might be affected by this change, please let me know.
        Hide
        shaie Shai Erera added a comment -

        If there are no objections, I'd like to commit that tomorrow.

        Show
        shaie Shai Erera added a comment - If there are no objections, I'd like to commit that tomorrow.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit deddc9b5c8d8c2859469583fa8b956be48efff82 in lucene-solr's branch refs/heads/master from Shai Erera
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=deddc9b ]

        SOLR-10349: Add totalTermFreq support to TermsComponent

        TermsComponent only returns docFreq information per requested term.
        This commit adds a terms.ttf parameter, which if set to true, will
        return both docFreq and totalTermFreq statistics for each requested
        term.

        Show
        jira-bot ASF subversion and git services added a comment - Commit deddc9b5c8d8c2859469583fa8b956be48efff82 in lucene-solr's branch refs/heads/master from Shai Erera [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=deddc9b ] SOLR-10349 : Add totalTermFreq support to TermsComponent TermsComponent only returns docFreq information per requested term. This commit adds a terms.ttf parameter, which if set to true, will return both docFreq and totalTermFreq statistics for each requested term.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit bcc36b9005afc5a36c1e9fc28ae6a9e5aedcd83d in lucene-solr's branch refs/heads/branch_6x from Shai Erera
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bcc36b9 ]

        SOLR-10349: Add totalTermFreq support to TermsComponent

        TermsComponent only returns docFreq information per requested term.
        This commit adds a terms.ttf parameter, which if set to true, will
        return both docFreq and totalTermFreq statistics for each requested
        term.

        Show
        jira-bot ASF subversion and git services added a comment - Commit bcc36b9005afc5a36c1e9fc28ae6a9e5aedcd83d in lucene-solr's branch refs/heads/branch_6x from Shai Erera [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bcc36b9 ] SOLR-10349 : Add totalTermFreq support to TermsComponent TermsComponent only returns docFreq information per requested term. This commit adds a terms.ttf parameter, which if set to true, will return both docFreq and totalTermFreq statistics for each requested term.
        Hide
        shaie Shai Erera added a comment -

        Pushed to master and branch_6x.

        Show
        shaie Shai Erera added a comment - Pushed to master and branch_6x.

          People

          • Assignee:
            shaie Shai Erera
            Reporter:
            shaie Shai Erera
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development