Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 6.3
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      The fetch() Streaming Expression wraps another expression and fetches additional fields for documents in batches. The fetch() expression will stream out the Tuples after the data has been fetched. Fields can be fetched from any SolrCloud collection.

      Sample syntax:

      daemon(
             update(collectionC, batchSize="100"
                    fetch(collectionB, 
                          topic(checkpoints, collectionA, q="*:*", fl="a,b,c", rows="50"),
                          fl="j,m,z",
                          on="a=j")))
             
      
      1. SOLR-9337.patch
        20 kB
        Joel Bernstein
      2. SOLR-9337.patch
        20 kB
        Joel Bernstein

        Activity

        Hide
        dpgove Dennis Gove added a comment -

        How does a fetch differ from an innerJoin? I guess it could if it read in a tuple from the source and then looked up its specific fields, but I dunno how performant that'd be.

        Show
        dpgove Dennis Gove added a comment - How does a fetch differ from an innerJoin? I guess it could if it read in a tuple from the source and then looked up its specific fields, but I dunno how performant that'd be.
        Hide
        joel.bernstein Joel Bernstein added a comment - - edited

        fetch works like this:

        1) read N tuples into memory
        2) use a query to fetch fields for the tuples read in step 1.
        3) stream the tuples out
        4) repeat steps 1-3 until the underlying stream is EOF

        This is essentially a nested loop join against the entire index.

        Mainly used when one side of the join is very small and you want to join it against the entire index.

        One main use case I have in mind is doing a graph query, fetching text fields for the node set that is returned, and then running the text classifier on the node set. This would combine graph queries and AI models to provide very intelligent recommendations.

        Show
        joel.bernstein Joel Bernstein added a comment - - edited fetch works like this: 1) read N tuples into memory 2) use a query to fetch fields for the tuples read in step 1. 3) stream the tuples out 4) repeat steps 1-3 until the underlying stream is EOF This is essentially a nested loop join against the entire index. Mainly used when one side of the join is very small and you want to join it against the entire index. One main use case I have in mind is doing a graph query, fetching text fields for the node set that is returned, and then running the text classifier on the node set. This would combine graph queries and AI models to provide very intelligent recommendations.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ee3f9e1e058ac4205140b909a85d43fdd715ddb7 in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee3f9e1 ]

        SOLR-9337: Add fetch Streaming Expression

        Show
        jira-bot ASF subversion and git services added a comment - Commit ee3f9e1e058ac4205140b909a85d43fdd715ddb7 in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ee3f9e1 ] SOLR-9337 : Add fetch Streaming Expression
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5836f4032fac975707c85e260d509ecd06c7f7e1 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5836f40 ]

        SOLR-9337: Add fetch Streaming Expression

        Show
        jira-bot ASF subversion and git services added a comment - Commit 5836f4032fac975707c85e260d509ecd06c7f7e1 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5836f40 ] SOLR-9337 : Add fetch Streaming Expression
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d69412bc676189600aed8b4cff2aad819526a5e2 in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d69412b ]

        SOLR-9337: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit d69412bc676189600aed8b4cff2aad819526a5e2 in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d69412b ] SOLR-9337 : Update CHANGES.txt
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ccc10fd5932fa5d830c3ecda86e85b4845bca863 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ccc10fd ]

        SOLR-9337: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit ccc10fd5932fa5d830c3ecda86e85b4845bca863 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ccc10fd ] SOLR-9337 : Update CHANGES.txt
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            Unassigned
            Reporter:
            joel.bernstein Joel Bernstein
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development