Solr
  1. Solr
  2. SOLR-7050

realtime get should internally load only fields specified in fl

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 5.1, 6.0
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      It is not always necessary to fetch all the stored fields in realtime get

      1. SOLR-7050.patch
        0.9 kB
        Noble Paul

        Activity

        Hide
        Yonik Seeley added a comment -

        Realtime get already honors the "fl" parameter.

        Show
        Yonik Seeley added a comment - Realtime get already honors the "fl" parameter.
        Hide
        Noble Paul added a comment -

        Sorry, In that case I tested it wrong.

        Show
        Noble Paul added a comment - Sorry, In that case I tested it wrong.
        Hide
        Noble Paul added a comment - - edited

        is it internally fetching all the fields ? because, when I debug I see all the fields populated in the SolrDocument object

        the toSolrDoc() method in RealtimeGetComponent does not seem to be aware of the fl param

        Show
        Noble Paul added a comment - - edited is it internally fetching all the fields ? because, when I debug I see all the fields populated in the SolrDocument object the toSolrDoc() method in RealtimeGetComponent does not seem to be aware of the fl param
        Hide
        Yonik Seeley added a comment - - edited

        This really doesn't have much to do with this issue (the API), but if you want to make the case of retrieving stored fields from the lucene index slightly more efficient, just replace this line:

               StoredDocument luceneDocument = searcher.doc(docid);
        

        With this line:

               StoredDocument luceneDocument = searcher.doc(docid, rsp.getReturnFields().getLuceneFieldNames());
        

        Also when converting the StoredDocument to SolrDocument, we would need to be careful not to touch the lazy fields (i.e. check if in rsp.getReturnFields().getLuceneFieldNames() first)

        Show
        Yonik Seeley added a comment - - edited This really doesn't have much to do with this issue (the API), but if you want to make the case of retrieving stored fields from the lucene index slightly more efficient, just replace this line: StoredDocument luceneDocument = searcher.doc(docid); With this line: StoredDocument luceneDocument = searcher.doc(docid, rsp.getReturnFields().getLuceneFieldNames()); Also when converting the StoredDocument to SolrDocument, we would need to be careful not to touch the lazy fields (i.e. check if in rsp.getReturnFields().getLuceneFieldNames() first)
        Hide
        Noble Paul added a comment -

        the reason why I raised the ticket was because the blob API does a realtime-get and it unnecessarily loads the huge binary field into memory when I'm just asking for an id

        Show
        Noble Paul added a comment - the reason why I raised the ticket was because the blob API does a realtime-get and it unnecessarily loads the huge binary field into memory when I'm just asking for an id
        Hide
        Yonik Seeley added a comment -

        Thank you for changing the title then.

        Show
        Yonik Seeley added a comment - Thank you for changing the title then.
        Hide
        Erick Erickson added a comment -

        Noble Paul

        Don't quite know if they're related to real-time-get, but the whole notion of decompressing everything to get anything is the subject of at least two other JIRAs you might be interested in:

        SOLR-5478 and SOLR-6810. I know this because I raised SOLR-6888 and David and Shalin pointed me to the other two

        Show
        Erick Erickson added a comment - Noble Paul Don't quite know if they're related to real-time-get, but the whole notion of decompressing everything to get anything is the subject of at least two other JIRAs you might be interested in: SOLR-5478 and SOLR-6810 . I know this because I raised SOLR-6888 and David and Shalin pointed me to the other two
        Hide
        Noble Paul added a comment -

        Erick Erickson They are not related.

        Show
        Noble Paul added a comment - Erick Erickson They are not related.
        Hide
        Erick Erickson added a comment -

        Yonik Seeley Thanks, not sure they were or weren't, covering all bases....

        Show
        Erick Erickson added a comment - Yonik Seeley Thanks, not sure they were or weren't, covering all bases....
        Hide
        ASF subversion and git services added a comment -

        Commit 1655540 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1655540 ]

        SOLR-7050: realtime get should internally load only fields specified in fl

        Show
        ASF subversion and git services added a comment - Commit 1655540 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1655540 ] SOLR-7050 : realtime get should internally load only fields specified in fl
        Hide
        ASF subversion and git services added a comment -

        Commit 1659912 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1659912 ]

        SOLR-7050: ealtime get should internally load only fields specified in fl

        Show
        ASF subversion and git services added a comment - Commit 1659912 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1659912 ] SOLR-7050 : ealtime get should internally load only fields specified in fl
        Hide
        Shalin Shekhar Mangar added a comment -

        Also when converting the StoredDocument to SolrDocument, we would need to be careful not to touch the lazy fields (i.e. check if in rsp.getReturnFields().getLuceneFieldNames() first)

        This optimization wasn't applied in the commits.

        Show
        Shalin Shekhar Mangar added a comment - Also when converting the StoredDocument to SolrDocument, we would need to be careful not to touch the lazy fields (i.e. check if in rsp.getReturnFields().getLuceneFieldNames() first) This optimization wasn't applied in the commits.
        Hide
        Noble Paul added a comment -
        Show
        Noble Paul added a comment - Shalin Shekhar Mangar correct.
        Hide
        Shalin Shekhar Mangar added a comment -

        This issue was committed but it is not mentioned in CHANGES.txt. Does anyone know in which release was it added?

        Show
        Shalin Shekhar Mangar added a comment - This issue was committed but it is not mentioned in CHANGES.txt. Does anyone know in which release was it added?
        Hide
        Steve Rowe added a comment - - edited

        The branch_5x commit (r1659912) came after the lucene_solr_5_0 branch was created (r1651551), and before the lucene_solr_5_1 branch was created (r1670247), so it must have been first released in 5.1.

        Show
        Steve Rowe added a comment - - edited The branch_5x commit ( r1659912 ) came after the lucene_solr_5_0 branch was created ( r1651551 ), and before the lucene_solr_5_1 branch was created ( r1670247 ), so it must have been first released in 5.1.
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Steve. Do you think we should add it to the CHANGES.txt in trunk and branch_5x at this later date?

        Show
        Shalin Shekhar Mangar added a comment - Thanks Steve. Do you think we should add it to the CHANGES.txt in trunk and branch_5x at this later date?
        Hide
        Steve Rowe added a comment -

        My vote is to add mistakenly omitted stuff in CHANGES.txt whenever we notice them. I think it's more important than keeping previous releases' notes exactly the same.

        Show
        Steve Rowe added a comment - My vote is to add mistakenly omitted stuff in CHANGES.txt whenever we notice them. I think it's more important than keeping previous releases' notes exactly the same.
        Hide
        ASF subversion and git services added a comment -

        Commit 1703869 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1703869 ]

        SOLR-7050: Adding entry to CHANGES.txt

        Show
        ASF subversion and git services added a comment - Commit 1703869 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1703869 ] SOLR-7050 : Adding entry to CHANGES.txt
        Hide
        ASF subversion and git services added a comment -

        Commit 1703875 from shalin@apache.org in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1703875 ]

        Added entry for SOLR-7110 and SOLR-7050 in the right places in CHANGES.txt

        Show
        ASF subversion and git services added a comment - Commit 1703875 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1703875 ] Added entry for SOLR-7110 and SOLR-7050 in the right places in CHANGES.txt

          People

          • Assignee:
            Noble Paul
            Reporter:
            Noble Paul
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development