Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8281

Add RollupMergeStream to Streaming API

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      The RollupMergeStream merges the aggregate results emitted by the RollupStream on worker nodes.

      This is designed to be used in conjunction with the HashJoinStream to perform rollup Aggregations on the joined Tuples. The HashJoinStream will require the tuples to be partitioned on the Join keys. To avoid needing to repartition on the group by fields for the RollupStream, we can perform a merge of the rolled up Tuples coming from the workers.

      The construct would like this:

      mergeRollup (...
                            parallel (...
                                          rollup (...
                                                      hashJoin (
                                                                        search(...),
                                                                        search(...),
                                                                        on="fieldA" 
                                                      )
                                           )
                               )
                     )
      

      The pseudo code above would push the hashJoin and rollup to the worker nodes. The emitted rolled up tuples would be merged by the mergeRollup.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jbernste Joel Bernstein
            jbernste Joel Bernstein
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment