Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10521

sort by string field of the nested child when searching with {!parent}

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.6, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      The idea is to integrate Lucene's ToParentBlockJoinSortField
      The approach to hookup it is a little bit tricky:
      sort={!childfield bjq=$q field=COLOR_s}desc sort=\childfield(COLOR_s,$q) desc
      the question no.1 wdyt about the syntax?
      internally it creates a function query with valueSource which produces ToParentBlockJoinSortField

      The main challenge is picking Solr field type from ToParentBlockJoinSortField, because as-is it's broken on mergeIds - ByteRefs need to be properly marshaled and unmarshalled by a field type from the child scope.

      1. SOLR-10521.patch
        25 kB
        Mikhail Khludnev
      2. SOLR-10521.patch
        26 kB
        Mikhail Khludnev
      3. SOLR-10521.patch
        25 kB
        Mikhail Khludnev
      4. SOLR-10521.patch
        26 kB
        Mikhail Khludnev
      5. SOLR-10521-6x.patch
        25 kB
        Mikhail Khludnev
      6. SOLR-10521-doc.patch
        1 kB
        Mikhail Khludnev
      7. SOLR-10521-raw.patch
        9 kB
        Mikhail Khludnev

        Issue Links

          Activity

          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Attaching the first raw scratch with a smoke only coverage. The significant gap is a lack of caching for bitsets. I also want to make QueryComponent change nicer.
          WDYT about it overall; regarding end user syntax; is there a way to fix marshalling in a better way (it's QueryComponent changes now)?

          Show
          mkhludnev Mikhail Khludnev added a comment - Attaching the first raw scratch with a smoke only coverage. The significant gap is a lack of caching for bitsets. I also want to make QueryComponent change nicer. WDYT about it overall; regarding end user syntax; is there a way to fix marshalling in a better way (it's QueryComponent changes now)?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Made a cool test. Caching and equality aren't addressed yet.
          Is there are any thoughts about the syntax?

          Show
          mkhludnev Mikhail Khludnev added a comment - Made a cool test. Caching and equality aren't addressed yet. Is there are any thoughts about the syntax?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-10521.patch done with equality testing, has a cool cloud test.

          I want to experiment with value source syntax, WDYT about sort=childfield(name,$q) asc?

          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-10521.patch done with equality testing, has a cool cloud test. I want to experiment with value source syntax, WDYT about sort=childfield(name,$q) asc ?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-10521.patch here we go!
          the valuesource syntax is sort=childfield(field,$bjq) asc or sort=childfield(field) asc assuming $q.
          The sad thing is that, I cant' improve QueryComponent change (it's brilliant already).
          So, far it can be only used for sorting. But can be extended to regular value source functionality in future. This might even work numerics, but I havent' checked it yet.
          Reviews and suggestions are urgently required!

          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-10521.patch here we go! the valuesource syntax is sort=childfield(field,$bjq) asc or sort=childfield(field) asc assuming $q . The sad thing is that, I cant' improve QueryComponent change (it's brilliant already). So, far it can be only used for sorting. But can be extended to regular value source functionality in future. This might even work numerics, but I havent' checked it yet. Reviews and suggestions are urgently required!
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          No feedback. It's ok. What about vetoes?

          Show
          mkhludnev Mikhail Khludnev added a comment - No feedback. It's ok. What about vetoes?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Here is the problem. QueryComponent needs to decide about a field type to marshal and unmarshal sort values.

          • it works as expected when sort=name asc. The correct field type is put to SortSpec.fields here. And then it's used to marshall ByteRef to String and back.
          • it works surprisingly fine when sorting by function like sort=sum(age, 42) asc. In this case SortSpecParsing puts null into SortSpec.fields. And then this null disables any marshaling that allows passing doubles and ints through javabin serialization (I care about SolrCloud), but it makes impossible to have a sort field returning BytesRef, because they are grabled with javabin. For example sort=field(name) asc doesn't work.
            Do you have an idea how to introduce sort=childfield(name) asc, without giving a lobotomy to myself QueryComponent?
          Show
          mkhludnev Mikhail Khludnev added a comment - Here is the problem. QueryComponent needs to decide about a field type to marshal and unmarshal sort values. it works as expected when sort=name asc . The correct field type is put to SortSpec.fields here . And then it's used to marshall ByteRef to String and back. it works surprisingly fine when sorting by function like sort=sum(age, 42) asc . In this case SortSpecParsing puts null into SortSpec.fields . And then this null disables any marshaling that allows passing doubles and ints through javabin serialization (I care about SolrCloud), but it makes impossible to have a sort field returning BytesRef , because they are grabled with javabin. For example sort=field(name) asc doesn't work. Do you have an idea how to introduce sort=childfield(name) asc , without giving a lobotomy to myself QueryComponent?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-10521.patch ByteRefs from Comparator are converted to byte[] and then unmarshalled back explicitly. Yet another one spaghetti. It seems like no-commit to me.

          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-10521.patch ByteRefs from Comparator are converted to byte[] and then unmarshalled back explicitly. Yet another one spaghetti. It seems like no-commit to me.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-10521.patch is a new hope. It wraps the true FieldComparator<ByteRefs> with a simple utf8ToString() converter. Let's take a one day timeout, and kick it off then.

          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-10521.patch is a new hope. It wraps the true FieldComparator<ByteRefs> with a simple utf8ToString() converter. Let's take a one day timeout, and kick it off then.
          Hide
          jove4015 Stephen Weiss added a comment -

          I'm thinking about how I would use this. And I do think it will probably come up. Most likely, I will want to both sort on a child (or grandchild, etc) field, and I will want to actually know what was in that field.

          So:

          a) It's important to include that extra $q parameter - don't assume there is only one level of children.
          b) Would it perhaps make more sense, and be easier on the QueryComponent, if you could just include arbitrary child / grandchild field values in the parent result set (perhaps, appearing as a multi-valued field?) If you did this, could you then simply sort on the resulting field? Ie, instead of changing the syntax of sort, allow child fields to be added to the response, and then sort them using the normal mechanism. I feel like it would be easier to modify the "fl" parameter because there's already a bracket syntax there (for example, for [docid]). Or maybe this already exists and I'm not aware of it? Something like fl=id,name,[!childfield=price_that_day object_type=sku_history price] and then sort=price_that_day desc? Actually, now that you mention it, I think we do have a ticket in the backlog that this feature would actually make feasible!

          Show
          jove4015 Stephen Weiss added a comment - I'm thinking about how I would use this. And I do think it will probably come up. Most likely, I will want to both sort on a child (or grandchild, etc) field, and I will want to actually know what was in that field. So: a) It's important to include that extra $q parameter - don't assume there is only one level of children. b) Would it perhaps make more sense, and be easier on the QueryComponent, if you could just include arbitrary child / grandchild field values in the parent result set (perhaps, appearing as a multi-valued field?) If you did this, could you then simply sort on the resulting field? Ie, instead of changing the syntax of sort, allow child fields to be added to the response, and then sort them using the normal mechanism. I feel like it would be easier to modify the "fl" parameter because there's already a bracket syntax there (for example, for [docid] ). Or maybe this already exists and I'm not aware of it? Something like fl=id,name, [!childfield=price_that_day object_type=sku_history price] and then sort=price_that_day desc? Actually, now that you mention it, I think we do have a ticket in the backlog that this feature would actually make feasible!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 816b806d8ac81940ccb9681c3b4f1d8727a395f7 in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=816b806 ]

          SOLR-10521: adding sort=childfield(field,$q) asc for

          {!parent}

          query.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 816b806d8ac81940ccb9681c3b4f1d8727a395f7 in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=816b806 ] SOLR-10521 : adding sort=childfield(field,$q) asc for {!parent} query.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-10521-6x.patch with stats instead of metrics

          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-10521-6x.patch with stats instead of metrics
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 189e50d9bcba4bd39e31fe6007ac56c1a882a61b in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=189e50d ]

          SOLR-10521: adding sort=childfield(field,$q) asc for

          {!parent}

          query.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 189e50d9bcba4bd39e31fe6007ac56c1a882a61b in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=189e50d ] SOLR-10521 : adding sort=childfield(field,$q) asc for {!parent} query.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Stephen Weiss,
          Thanks for feedback.

          Would it perhaps make more sense, and be easier on the QueryComponent

          I wouldn't touch it ever, too many stuff has been injected into it already. However, solid approach always seems promising. Perhaps, it can be done in the dedicated component, and might be considered as a step in SOLR-10144

          I feel like it would be easier to modify the "fl" parameter

          right, we already have [child], and [subquery] (I'm sorry about that). The problem is that sort, fl has completely different scope. Just one case, we might fl non-indexed child field, and sort by non-stored child field too.

          Show
          mkhludnev Mikhail Khludnev added a comment - Stephen Weiss , Thanks for feedback. Would it perhaps make more sense, and be easier on the QueryComponent I wouldn't touch it ever, too many stuff has been injected into it already. However, solid approach always seems promising. Perhaps, it can be done in the dedicated component, and might be considered as a step in SOLR-10144 I feel like it would be easier to modify the "fl" parameter right, we already have [child], and [subquery] (I'm sorry about that). The problem is that sort , fl has completely different scope. Just one case, we might fl non-indexed child field, and sort by non-stored child field too.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Cassandra Targett, would you mind to look at SOLR-10521-doc.patch?

          Show
          mkhludnev Mikhail Khludnev added a comment - Cassandra Targett , would you mind to look at SOLR-10521-doc.patch ?
          Hide
          ctargett Cassandra Targett added a comment -

          Mikhail Khludnev - Yes it looks good. +1, thanks.

          Show
          ctargett Cassandra Targett added a comment - Mikhail Khludnev - Yes it looks good. +1, thanks.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9b6b3e6260ed1ac53a80cc8d6d9e88635f0c5c16 in lucene-solr's branch refs/heads/branch_6_6 from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9b6b3e6 ]

          SOLR-10521: documenting sort=childfield(field) asc

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9b6b3e6260ed1ac53a80cc8d6d9e88635f0c5c16 in lucene-solr's branch refs/heads/branch_6_6 from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9b6b3e6 ] SOLR-10521 : documenting sort=childfield(field) asc
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a4bd80054fbd2129f4bea77c3bdb1e7e82da9975 in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a4bd800 ]

          SOLR-10521: documenting sort=childfield(field) asc

          Show
          jira-bot ASF subversion and git services added a comment - Commit a4bd80054fbd2129f4bea77c3bdb1e7e82da9975 in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a4bd800 ] SOLR-10521 : documenting sort=childfield(field) asc
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7646f91097c54890348b2f9dda9cf4e554c3f77d in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7646f91 ]

          SOLR-10521: documenting sort=childfield(field) asc

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7646f91097c54890348b2f9dda9cf4e554c3f77d in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7646f91 ] SOLR-10521 : documenting sort=childfield(field) asc
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Cassandra Targett, I've checked the generated guide. Formatting around childfield() has minor issues, I'm not sure how to address them. fyi.

          Show
          mkhludnev Mikhail Khludnev added a comment - Cassandra Targett , I've checked the generated guide. Formatting around childfield() has minor issues, I'm not sure how to address them. fyi.
          Hide
          ctargett Cassandra Targett added a comment -

          Mikhail Khludnev, yes, I also noticed those as I was looking over the generated guide late yesterday afternoon. Lots of other examples in that page also had issues from the conversion. I have fixed them locally, but haven't pushed it yet, I will do that soon with a group of minor typos I've found on other pages also.

          Show
          ctargett Cassandra Targett added a comment - Mikhail Khludnev , yes, I also noticed those as I was looking over the generated guide late yesterday afternoon. Lots of other examples in that page also had issues from the conversion. I have fixed them locally, but haven't pushed it yet, I will do that soon with a group of minor typos I've found on other pages also.

            People

            • Assignee:
              Unassigned
              Reporter:
              mkhludnev Mikhail Khludnev
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development