Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8577

Add AlertStream and ModelStream to the Streaming API

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The AlertStream will return the top N "new" documents for a query from a SolrCloud collection. The AlertStream will track the highest version numbers from each shard and use these as checkpoints to determine new content.

      The DaemonStream (SOLR-8550) can be used to create "live" alerts that run at intervals. Sample syntax:

      daemon(alert(collection1, q="hello", n="20"), runInterval="2000")
      

      The DaemonStream can be installed in a SolrCloud worker node where it can llive and send out alerts.

      AI Models

      The AlertStream will also accept an optional ModelStream which will apply a machine learning model to the alert. For example:

      alert(collection1, q="hello", n="20", model(collection2, id="model1"))
      

      The ModelStream will return a machine learning model saved in a SolrCloud collection. Function queries for different model types will be developed so the models can be applied in the re-ranker or as a sort.

      Taking action

      Custom decorator streams can be developed that take actions based on the AI driven alerts. For example the pseudo code below would run the function someAction on the Tuples emitted by the AlertStream.

      daemon(someAction(alert(...)))
      

      Learning

      While some SolrCloud worker collections are alerting and taking action, other worker collections can be learning models which can be applied for alerting. For example:

      daemon(update(logit()))
      

      The pseudo code above calls the LogitStream (SOLR-8492) which would learn a Logistic Regression model and flow the model into a SolrCloud collection. The model can then be used for alerting and taking action on new data as it enters the system.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                joel.bernstein Joel Bernstein
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: