Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Implemented
    • Affects Version/s: 6.0
    • Fix Version/s: 6.0
    • Component/s: SolrJ
    • Labels:

      Description

      Enhances MergeStream to support merging N streams. This was previously limited to merging just two streams but with this enhancement it can now accept any number of streams to merge.

      Based on the comparator, if more than one stream could provide the next value then the selected value will follow the order of the streams as they appear in the expression or were added to the MergeStream object.

      merge(
        search(collection1, q="id:(0 4)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
        search(collection1, q="id:(1)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
        search(collection1, q="id:(2)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
        on="a_f asc"
      )
      

        Issue Links

          Activity

          Hide
          dpgove Dennis Gove added a comment -

          Any thoughts on the feature or attached patch?

          Show
          dpgove Dennis Gove added a comment - Any thoughts on the feature or attached patch?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I think the feature looks great. I'm just wrapping up SOLR-8086, which will pretty much clear my plate.

          Let's create an umbrella ticket for Streaming and SQL issues so we don't lose track of them. We can link this ticket and other outstanding Streaming and SQL tickets to the umbrella ticket. The umbrella ticket can also be a place to discuss the road map.

          Show
          joel.bernstein Joel Bernstein added a comment - I think the feature looks great. I'm just wrapping up SOLR-8086 , which will pretty much clear my plate. Let's create an umbrella ticket for Streaming and SQL issues so we don't lose track of them. We can link this ticket and other outstanding Streaming and SQL tickets to the umbrella ticket. The umbrella ticket can also be a place to discuss the road map.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          I just created the umbrella ticket and linked it. If you have other tickets out there feel free to link them to SOLR-8125.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited I just created the umbrella ticket and linked it. If you have other tickets out there feel free to link them to SOLR-8125 .
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Just reviewing the implementation for this ticket. Rather then having the MergeStream take N streams we could have a MultiMergeStream that under the covers would loop over the streams and create compounded MergeStreams.

          For example merging streams a,b and c could be expressed like this inside of the MultiMergeStream:

          stream = new MergeStream(new MergeStream(a, c, comp), b, comp);
          

          There's something kind of nice about this compounding of MergeStreams.

          The Expression would like this:

          multiMerge(
            search(collection1, q="id:(0 4)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
            search(collection1, q="id:(1)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
            search(collection1, q="id:(2)", fl="id,a_s,a_i,a_f", sort="a_f asc, a_s asc"),
            on="a_f asc"
          )
          
          Show
          joel.bernstein Joel Bernstein added a comment - - edited Just reviewing the implementation for this ticket. Rather then having the MergeStream take N streams we could have a MultiMergeStream that under the covers would loop over the streams and create compounded MergeStreams. For example merging streams a,b and c could be expressed like this inside of the MultiMergeStream: stream = new MergeStream( new MergeStream(a, c, comp), b, comp); There's something kind of nice about this compounding of MergeStreams. The Expression would like this: multiMerge( search(collection1, q= "id:(0 4)" , fl= "id,a_s,a_i,a_f" , sort= "a_f asc, a_s asc" ), search(collection1, q= "id:(1)" , fl= "id,a_s,a_i,a_f" , sort= "a_f asc, a_s asc" ), search(collection1, q= "id:(2)" , fl= "id,a_s,a_i,a_f" , sort= "a_f asc, a_s asc" ), on= "a_f asc" )
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1713190 from [~dgove] in branch 'dev/trunk'
          [ https://svn.apache.org/r1713190 ]

          SOLR-7938: MergeStream now supports merging more than 2 streams together

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1713190 from [~dgove] in branch 'dev/trunk' [ https://svn.apache.org/r1713190 ] SOLR-7938 : MergeStream now supports merging more than 2 streams together
          Hide
          dpgove Dennis Gove added a comment -

          (just committed this, looks like we were working simultaneously)

          I'm not sure I see the benefit of a MultiMergeStream over a MergeStream supporting N streams. As a user I might wonder why I need to provide multiMerge instead of merge just because I want to add a 3rd substream. I dunno that the user experience would benefit.

          Show
          dpgove Dennis Gove added a comment - (just committed this, looks like we were working simultaneously) I'm not sure I see the benefit of a MultiMergeStream over a MergeStream supporting N streams. As a user I might wonder why I need to provide multiMerge instead of merge just because I want to add a 3rd substream. I dunno that the user experience would benefit.
          Hide
          dpgove Dennis Gove added a comment - - edited

          Still need to update https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions with the updated functionality.

          The page at that link is marked as "This Unreleased Guide Will Cover Apache Solr 5.4". As this patch has only been added to trunk at this point I believe that page shouldn't actually be updated.

          I did look at the backporting of this to branch_5x but it depends on a new interface added in SOLR-7707 which has not been backported. Going to investigate splitting that patch up into pieces which can be backported to branch_5x.

          Show
          dpgove Dennis Gove added a comment - - edited Still need to update https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions with the updated functionality. The page at that link is marked as "This Unreleased Guide Will Cover Apache Solr 5.4". As this patch has only been added to trunk at this point I believe that page shouldn't actually be updated. I did look at the backporting of this to branch_5x but it depends on a new interface added in SOLR-7707 which has not been backported. Going to investigate splitting that patch up into pieces which can be backported to branch_5x.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Sounds good, congrats on your first commit!

          Show
          joel.bernstein Joel Bernstein added a comment - Sounds good, congrats on your first commit!

            People

            • Assignee:
              dpgove Dennis Gove
              Reporter:
              dpgove Dennis Gove
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development