Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7554

Add checks in Streams for incoming stream order

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 5.2, 6.0
    • Fix Version/s: 6.0
    • Component/s: SolrJ
    • Labels:

      Description

      Most Streams built on top of other streams require that their incoming stream(s) be ordered in a complimentary way to how this stream is expected to output its results.

      For example, if a MergeStream is merging two streams on "fieldA asc, fieldB desc", then both its incoming streams must be ordered in a similar way. That said, the incoming stream could be ordered more strictly, ie "fieldA asc, fieldB desc, fieldC asc" but as long as the the comparator used in the MergeStream can be "derived from" the incoming stream's comparator then we are good to go.

      Some comparator A can be "derived from" some other comparator B iff the fields and their order in A is equal to the first fields and their order in B. For example, "fieldA asc, fieldB dec" can be derived from "fieldA asc, fieldB desc, fieldC asc, fieldD asc" but cannot be derived from "field A asc".

      This patch is to add this validation support. It requires changes to Comparators, Equalitors, most Streams, and related tests. It adds a way to compare Comparators and Equalitors and in the end is one more required piece before we can add support for Join streams.

      It is dependent on SOLR-7513 and SOLR-7528. Other dependencies it has have already been committed to trunk and the 5.2 branch.

      It does not change any interfaces to code already released (5.1 and below). It does change interfaces to code in trunk and 5.2.

      1. SOLR-7554.patch
        83 kB
        Joel Bernstein
      2. SOLR-7554.patch
        80 kB
        Joel Bernstein
      3. SOLR-7554.patch
        78 kB
        Dennis Gove

        Activity

        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        It does not change any interfaces to code already released (5.1 and below).

        Just a general note that we do have more ability to change things across releases if it's marked as experimental. I'd recommend anything with a complex API to be designated as experimental at first, to give time for improvements after more people have had a chance to look / use it.

        Show
        yseeley@gmail.com Yonik Seeley added a comment - It does not change any interfaces to code already released (5.1 and below). Just a general note that we do have more ability to change things across releases if it's marked as experimental. I'd recommend anything with a complex API to be designated as experimental at first, to give time for improvements after more people have had a chance to look / use it.
        Hide
        joel.bernstein Joel Bernstein added a comment - - edited

        New patch built with svn diff. Streaming tests are passing. Looks like all this ticket needs is a few tests that demonstrate Streams throwing an exception when the sorts don't line up properly. I will add those tests to the patch shortly.

        Thanks [~dgove1], this is an important patch which should eliminate hard to recognize bugs that will appear if/when sorts are incompatible.

        Show
        joel.bernstein Joel Bernstein added a comment - - edited New patch built with svn diff. Streaming tests are passing. Looks like all this ticket needs is a few tests that demonstrate Streams throwing an exception when the sorts don't line up properly. I will add those tests to the patch shortly. Thanks [~dgove1] , this is an important patch which should eliminate hard to recognize bugs that will appear if/when sorts are incompatible.
        Hide
        joel.bernstein Joel Bernstein added a comment - - edited

        Added tests that ensure the proper exception is being thrown for the MergeStream, UniqueStream and ReducerStream.

        Didn't add the parallel tests yet as Exceptions are not being propagated from the workers yet. Those tests can be added in another ticket after exception propagation is put in place for parallel requests.

        Show
        joel.bernstein Joel Bernstein added a comment - - edited Added tests that ensure the proper exception is being thrown for the MergeStream, UniqueStream and ReducerStream. Didn't add the parallel tests yet as Exceptions are not being propagated from the workers yet. Those tests can be added in another ticket after exception propagation is put in place for parallel requests.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1687258 from Joel Bernstein in branch 'dev/trunk'
        [ https://svn.apache.org/r1687258 ]

        SOLR-7554: Add checks in Streams for incoming stream order

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1687258 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1687258 ] SOLR-7554 : Add checks in Streams for incoming stream order
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1692327 from Joel Bernstein in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1692327 ]

        SOLR-7554: Add checks in Streams for incoming stream order

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1692327 from Joel Bernstein in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692327 ] SOLR-7554 : Add checks in Streams for incoming stream order
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1692329 from Joel Bernstein in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1692329 ]

        SOLR-7554: Reverting 5x commit

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1692329 from Joel Bernstein in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692329 ] SOLR-7554 : Reverting 5x commit
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1694908 from Joel Bernstein in branch 'dev/trunk'
        [ https://svn.apache.org/r1694908 ]

        SOLR-7554: Updated CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1694908 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1694908 ] SOLR-7554 : Updated CHANGES.txt
        Hide
        joel.bernstein Joel Bernstein added a comment -

        Release with Solr 6

        Show
        joel.bernstein Joel Bernstein added a comment - Release with Solr 6

          People

          • Assignee:
            Unassigned
            Reporter:
            dpgove Dennis Gove
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development